]> git.saurik.com Git - apple/xnu.git/blame - iokit/Kernel/IOHibernateIO.cpp
projects / apple / xnu.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
xnu-1504.7.4.tar.gz
[apple/xnu.git] / iokit / Kernel / IOHibernateIO.cpp
CommitLineData
3a60a9f5 1/*
b0d623f7 2 * Copyright (c) 2004-2008 Apple Computer, Inc. All rights reserved.
3a60a9f5 3 *
2d21ac55 4 * @APPLE_OSREFERENCE_LICENSE_HEADER_START@
3a60a9f5 5 *
2d21ac55
A		 6 * This file contains Original Code and/or Modifications of Original Code
7 * as defined in and that are subject to the Apple Public Source License
8 * Version 2.0 (the 'License'). You may not use this file except in
9 * compliance with the License. The rights granted to you under the License
10 * may not be used to create, or enable the creation or redistribution of,
11 * unlawful or unlicensed copies of an Apple operating system, or to
12 * circumvent, violate, or enable the circumvention or violation of, any
13 * terms of an Apple operating system software license agreement.
8f6c56a5 14 *
2d21ac55
A		 15 * Please obtain a copy of the License at
16 * http://www.opensource.apple.com/apsl/ and read it before using this file.
17 *
18 * The Original Code and all software distributed under the License are
19 * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
8f6c56a5
A		 20 * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
21 * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
2d21ac55
A		 22 * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
23 * Please see the License for the specific language governing rights and
24 * limitations under the License.
8f6c56a5 25 *
2d21ac55 26 * @APPLE_OSREFERENCE_LICENSE_HEADER_END@
3a60a9f5
A		 27 */
28
29
30/*
31
32Sleep:
33
34- PMRootDomain calls IOHibernateSystemSleep() before system sleep
35(devices awake, normal execution context)
36- IOHibernateSystemSleep opens the hibernation file (or partition) at the bsd level,
37 grabs its extents and searches for a polling driver willing to work with that IOMedia.
38 The BSD code makes an ioctl to the storage driver to get the partition base offset to
39 the disk, and other ioctls to get the transfer constraints
40 If successful, the file is written to make sure its initially not bootable (in case of
41 later failure) and nvram set to point to the first block of the file. (Has to be done
42 here so blocking is possible in nvram support).
43 hibernate_setup() in osfmk is called to allocate page bitmaps for all dram, and
44 page out any pages it wants to (currently zero, but probably some percentage of memory).
45 Its assumed just allocating pages will cause the VM system to naturally select the best
46 pages for eviction. It also copies processor flags needed for the restore path and sets
47 a flag in the boot processor proc info.
48 gIOHibernateState = kIOHibernateStateHibernating.
49- Regular sleep progresses - some drivers may inspect the root domain property
50 kIOHibernateStateKey to modify behavior. The platform driver saves state to memory
51 as usual but leaves motherboard I/O on.
52- Eventually the platform calls ml_ppc_sleep() in the shutdown context on the last cpu,
53 at which point memory is ready to be saved. mapping_hibernate_flush() is called to get
54 all ppc RC bits out of the hash table and caches into the mapping structures.
55- hibernate_write_image() is called (still in shutdown context, no blocking or preemption).
56 hibernate_page_list_setall() is called to get a bitmap of dram pages that need to be saved.
57 All pages are assumed to be saved (as part of the wired image) unless explicitly subtracted
58 by hibernate_page_list_setall(), avoiding having to find arch dependent low level bits.
59 The image header and block list are written. The header includes the second file extent so
60 only the header block is needed to read the file, regardless of filesystem.
61 The kernel section "__HIB" is written uncompressed to the image. This section of code and data
62 (only) is used to decompress the image during wake/boot.
63 Some additional pages are removed from the bitmaps - the buffers used for hibernation.
64 The bitmaps are written to the image.
65 More areas are removed from the bitmaps (after they have been written to the image) - the
66 section "__HIB" pages and interrupt stack.
67 Each wired page is compressed and written and then each non-wired page. Compression and
68 disk writes are in parallel.
69 The image header is written to the start of the file and the polling driver closed.
70 The machine powers down (or sleeps).
71
72Boot/Wake:
73
74- BootX sees the boot-image nvram variable containing the device and block number of the image,
75 reads the header and if the signature is correct proceeds. The boot-image variable is cleared.
76- BootX reads the portion of the image used for wired pages, to memory. Its assumed this will fit
77 in the OF memory environment, and the image is decrypted. There is no decompression in BootX,
78 that is in the kernel's __HIB section.
79- BootX copies the "__HIB" section to its correct position in memory, quiesces and calls its entry
80 hibernate_kernel_entrypoint(), passing the location of the image in memory. Translation is off,
81 only code & data in that section is safe to call since all the other wired pages are still
82 compressed in the image.
83- hibernate_kernel_entrypoint() removes pages occupied by the raw image from the page bitmaps.
84 It uses the bitmaps to work out which pages can be uncompressed from the image to their final
85 location directly, and copies those that can't to interim free pages. When the image has been
86 completed, the copies are uncompressed, overwriting the wired image pages.
87 hibernate_restore_phys_page() (in osfmk since its arch dependent, but part of the "__HIB" section)
88 is used to get pages into place for 64bit.
89- the reset vector is called (at least on ppc), the kernel proceeds on a normal wake, with some
90 changes conditional on the per proc flag - before VM is turned on the boot cpu, all mappings
91 are removed from the software strutures, and the hash table is reinitialized.
92- After the platform CPU init code is called, hibernate_machine_init() is called to restore the rest
93 of memory, using the polled mode driver, before other threads can run or any devices are turned on.
94 This reduces the memory usage for BootX and allows decompression in parallel with disk reads,
95 for the remaining non wired pages.
96- The polling driver is closed down and regular wake proceeds. When the kernel calls iokit to wake
97 (normal execution context) hibernate_teardown() in osmfk is called to release any memory, the file
98 is closed via bsd.
99
100Polled Mode I/O:
101
102IOHibernateSystemSleep() finds a polled mode interface to the ATA controller via a property in the
103registry, specifying an object of calls IOPolledInterface.
104
105Before the system goes to sleep it searches from the IOMedia object (could be a filesystem or
106partition) that the image is going to live, looking for polled interface properties. If it finds
107one the IOMedia object is passed to a "probe" call for the interface to accept or reject. All the
108interfaces found are kept in an ordered list.
109
110There is an Open/Close pair of calls made to each of the interfaces at various stages since there are
111few different contexts things happen in:
112
113- there is an Open/Close (Preflight) made before any part of the system has slept (I/O is all
114up and running) and after wake - this is safe to allocate memory and do anything. The device
115ignores sleep requests from that point since its a waste of time if it goes to sleep and
116immediately wakes back up for the image write.
117
118- there is an Open/Close (BeforeSleep) pair made around the image write operations that happen
119immediately before sleep. These can't block or allocate memory - the I/O system is asleep apart
120from the low level bits (motherboard I/O etc). There is only one thread running. The close can be
121used to flush and set the disk to sleep.
122
123- there is an Open/Close (AfterSleep) pair made around the image read operations that happen
124immediately after sleep. These can't block or allocate memory. This is happening after the platform
125expert has woken the low level bits of the system, but most of the I/O system has not. There is only
126one thread running.
127
128For the actual I/O, all the ops are with respect to a single IOMemoryDescriptor that was passed
129(prepared) to the Preflight Open() call. There is a read/write op, buffer offset to the IOMD for
130the data, an offset to the disk and length (block aligned 64 bit numbers), and completion callback.
131Each I/O is async but only one is ever outstanding. The polled interface has a checkForWork call
132that is called for the hardware to check for events, and complete the I/O via the callback.
133The hibernate path uses the same transfer constraints the regular cluster I/O path in BSD uses
134to restrict I/O ops.
135*/
136
137#include <sys/systm.h>
138
139#include <IOKit/IOWorkLoop.h>
140#include <IOKit/IOCommandGate.h>
141#include <IOKit/IOTimerEventSource.h>
142#include <IOKit/IOPlatformExpert.h>
143#include <IOKit/IOKitDebug.h>
144#include <IOKit/IOTimeStamp.h>
145#include <IOKit/pwr_mgt/RootDomain.h>
146#include <IOKit/pwr_mgt/IOPMPrivate.h>
147#include <IOKit/IOMessage.h>
148#include <IOKit/IODeviceTreeSupport.h>
149#include <IOKit/IOBSD.h>
150#include "RootDomainUserClient.h"
151#include <IOKit/pwr_mgt/IOPowerConnection.h>
152#include "IOPMPowerStateQueue.h"
153#include <IOKit/IOBufferMemoryDescriptor.h>
154#include <crypto/aes.h>
155
156#include <sys/uio.h>
157#include <sys/conf.h>
158#include <sys/stat.h>
159#include <sys/fcntl.h> // (FWRITE, ...)
3a60a9f5 160#include <sys/sysctl.h>
3a60a9f5
A		 161
162#include <IOKit/IOHibernatePrivate.h>
163#include <IOKit/IOPolledInterface.h>
164#include <IOKit/IONVRAM.h>
165#include "IOHibernateInternal.h"
166#include "WKdm.h"
167#include "IOKitKernelInternal.h"
168
169/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
170
3a60a9f5
A		 171extern uint32_t gIOHibernateState;
172uint32_t gIOHibernateMode;
173static char gIOHibernateBootSignature[256+1];
174static char gIOHibernateFilename[MAXPATHLEN+1];
175static uint32_t gIOHibernateFreeRatio = 0; // free page target (percent)
176uint32_t gIOHibernateFreeTime = 0*1000; // max time to spend freeing pages (ms)
177
178static IODTNVRAM * gIOOptionsEntry;
179static IORegistryEntry * gIOChosenEntry;
b0d623f7 180#if defined(__i386__) || defined(__x86_64__)
0c530ab8
A		 181static const OSSymbol * gIOCreateEFIDevicePathSymbol;
182#endif
3a60a9f5
A		 183
184static IOPolledFileIOVars gFileVars;
185static IOHibernateVars gIOHibernateVars;
186static struct kern_direct_file_io_ref_t * gIOHibernateFileRef;
187static hibernate_cryptvars_t gIOHibernateCryptWakeContext;
188
189/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
190
191enum { kXPRamAudioVolume = 8 };
192enum { kDefaultIOSize = 128 * 1024 };
193enum { kVideoMapSize = 32 * 1024 * 1024 };
194
195#ifndef kIOMediaPreferredBlockSizeKey
196#define kIOMediaPreferredBlockSizeKey "Preferred Block Size"
197#endif
198
199#ifndef kIOBootPathKey
200#define kIOBootPathKey "bootpath"
201#endif
202#ifndef kIOSelectedBootDeviceKey
203#define kIOSelectedBootDeviceKey "boot-device"
204#endif
205
206
207enum { kIOHibernateMinPollersNeeded = 2 };
208
209/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
210
211// copy from phys addr to MD
212
213static IOReturn
214IOMemoryDescriptorWriteFromPhysical(IOMemoryDescriptor * md,
215 IOByteCount offset, addr64_t bytes, IOByteCount length)
216{
217 addr64_t srcAddr = bytes;
218 IOByteCount remaining;
219
220 remaining = length = min(length, md->getLength() - offset);
221 while (remaining) { // (process another target segment?)
222 addr64_t dstAddr64;
223 IOByteCount dstLen;
224
b0d623f7 225 dstAddr64 = md->getPhysicalSegment(offset, &dstLen, kIOMemoryMapperNone);
3a60a9f5
A		 226 if (!dstAddr64)
227 break;
228
229 // Clip segment length to remaining
230 if (dstLen > remaining)
231 dstLen = remaining;
232
233#if 1
234 bcopy_phys(srcAddr, dstAddr64, dstLen);
235#else
236 copypv(srcAddr, dstAddr64, dstLen,
237 cppvPsnk | cppvFsnk | cppvNoRefSrc | cppvNoModSnk | cppvKmap);
238#endif
239 srcAddr += dstLen;
240 offset += dstLen;
241 remaining -= dstLen;
242 }
243
244 assert(!remaining);
245
246 return remaining ? kIOReturnUnderrun : kIOReturnSuccess;
247}
248
249// copy from MD to phys addr
250
251static IOReturn
252IOMemoryDescriptorReadToPhysical(IOMemoryDescriptor * md,
253 IOByteCount offset, addr64_t bytes, IOByteCount length)
254{
255 addr64_t dstAddr = bytes;
256 IOByteCount remaining;
257
258 remaining = length = min(length, md->getLength() - offset);
259 while (remaining) { // (process another target segment?)
260 addr64_t srcAddr64;
261 IOByteCount dstLen;
262
b0d623f7 263 srcAddr64 = md->getPhysicalSegment(offset, &dstLen, kIOMemoryMapperNone);
3a60a9f5
A		 264 if (!srcAddr64)
265 break;
266
267 // Clip segment length to remaining
268 if (dstLen > remaining)
269 dstLen = remaining;
270
271#if 1
272 bcopy_phys(srcAddr64, dstAddr, dstLen);
273#else
274 copypv(srcAddr, dstAddr64, dstLen,
275 cppvPsnk | cppvFsnk | cppvNoRefSrc | cppvNoModSnk | cppvKmap);
276#endif
277 dstAddr += dstLen;
278 offset += dstLen;
279 remaining -= dstLen;
280 }
281
282 assert(!remaining);
283
284 return remaining ? kIOReturnUnderrun : kIOReturnSuccess;
285}
286
287/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
288
289void
290hibernate_set_page_state(hibernate_page_list_t * page_list, hibernate_page_list_t * page_list_wired,
291 vm_offset_t ppnum, vm_offset_t count, uint32_t kind)
292{
293 count += ppnum;
294 switch (kind)
295 {
296 case kIOHibernatePageStateUnwiredSave:
297 // unwired save
298 for (; ppnum < count; ppnum++)
299 {
300 hibernate_page_bitset(page_list, FALSE, ppnum);
301 hibernate_page_bitset(page_list_wired, TRUE, ppnum);
302 }
303 break;
304 case kIOHibernatePageStateWiredSave:
305 // wired save
306 for (; ppnum < count; ppnum++)
307 {
308 hibernate_page_bitset(page_list, FALSE, ppnum);
309 hibernate_page_bitset(page_list_wired, FALSE, ppnum);
310 }
311 break;
312 case kIOHibernatePageStateFree:
313 // free page
314 for (; ppnum < count; ppnum++)
315 {
316 hibernate_page_bitset(page_list, TRUE, ppnum);
317 hibernate_page_bitset(page_list_wired, TRUE, ppnum);
318 }
319 break;
320 default:
321 panic("hibernate_set_page_state");
322 }
323}
324
325static vm_offset_t
0c530ab8 326hibernate_page_list_iterate(hibernate_page_list_t * list, vm_offset_t * pPage)
3a60a9f5 327{
0c530ab8
A		 328 uint32_t page = *pPage;
329 uint32_t count;
330 hibernate_bitmap_t * bitmap;
21362eb3 331
0c530ab8
A		 332 while ((bitmap = hibernate_page_bitmap_pin(list, &page)))
333 {
334 count = hibernate_page_bitmap_count(bitmap, TRUE, page);
335 if (!count)
336 break;
337 page += count;
338 if (page <= bitmap->last_page)
339 break;
340 }
6601e61a 341
0c530ab8
A		 342 *pPage = page;
343 if (bitmap)
344 count = hibernate_page_bitmap_count(bitmap, FALSE, page);
345 else
346 count = 0;
3a60a9f5
A		 347
348 return (count);
349}
350
351/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
352
353static IOReturn
354IOHibernatePollerProbe(IOPolledFileIOVars * vars, IOService * target)
355{
356 IOReturn err = kIOReturnError;
357 int32_t idx;
358 IOPolledInterface * poller;
359
360 for (idx = vars->pollers->getCount() - 1; idx >= 0; idx--)
361 {
362 poller = (IOPolledInterface *) vars->pollers->getObject(idx);
363 err = poller->probe(target);
364 if (err)
365 {
366 HIBLOG("IOPolledInterface::probe[%d] 0x%x\n", idx, err);
367 break;
368 }
369 }
370
371 return (err);
372}
373
374static IOReturn
375IOHibernatePollerOpen(IOPolledFileIOVars * vars, uint32_t state, IOMemoryDescriptor * md)
376{
377 IOReturn err = kIOReturnError;
378 int32_t idx;
379 IOPolledInterface * poller;
380
381 for (idx = vars->pollers->getCount() - 1; idx >= 0; idx--)
382 {
383 poller = (IOPolledInterface *) vars->pollers->getObject(idx);
384 err = poller->open(state, md);
385 if (err)
386 {
387 HIBLOG("IOPolledInterface::open[%d] 0x%x\n", idx, err);
388 break;
389 }
390 }
391
392 return (err);
393}
394
395static IOReturn
396IOHibernatePollerClose(IOPolledFileIOVars * vars, uint32_t state)
397{
398 IOReturn err = kIOReturnError;
399 int32_t idx;
400 IOPolledInterface * poller;
401
402 for (idx = 0;
403 (poller = (IOPolledInterface *) vars->pollers->getObject(idx));
404 idx++)
405 {
406 err = poller->close(state);
407 if (err)
408 HIBLOG("IOPolledInterface::close[%d] 0x%x\n", idx, err);
409 }
410
411 return (err);
412}
413
414static void
415IOHibernatePollerIOComplete(void * target,
416 void * parameter,
417 IOReturn status,
418 UInt64 actualByteCount)
419{
420 IOPolledFileIOVars * vars = (IOPolledFileIOVars *) parameter;
421
422 vars->ioStatus = status;
423}
424
425static IOReturn
426IOHibernatePollerIO(IOPolledFileIOVars * vars,
427 uint32_t operation, uint32_t bufferOffset,
428 uint64_t deviceOffset, uint64_t length)
429{
430
431 IOReturn err = kIOReturnError;
432 IOPolledInterface * poller;
433 IOPolledCompletion completion;
434
435 completion.target = 0;
436 completion.action = &IOHibernatePollerIOComplete;
437 completion.parameter = vars;
438
439 vars->ioStatus = -1;
440
441 poller = (IOPolledInterface *) vars->pollers->getObject(0);
442 err = poller->startIO(operation, bufferOffset, deviceOffset + vars->block0, length, completion);
443 if (err)
444 HIBLOG("IOPolledInterface::startIO[%d] 0x%x\n", 0, err);
445
446 return (err);
447}
448
449static IOReturn
2d21ac55 450IOHibernatePollerIODone(IOPolledFileIOVars * vars, bool abortable)
3a60a9f5 451{
2d21ac55
A		 452 IOReturn err = kIOReturnSuccess;
453 int32_t idx = 0;
3a60a9f5
A		 454 IOPolledInterface * poller;
455
456 while (-1 == vars->ioStatus)
457 {
2d21ac55
A		 458 for (idx = 0;
459 (poller = (IOPolledInterface *) vars->pollers->getObject(idx));
3a60a9f5
A		 460 idx++)
461 {
2d21ac55
A		 462 IOReturn newErr;
463 newErr = poller->checkForWork();
464 if ((newErr == kIOReturnAborted) && !abortable)
465 newErr = kIOReturnSuccess;
466 if (kIOReturnSuccess == err)
467 err = newErr;
3a60a9f5
A		 468 }
469 }
470
2d21ac55
A		 471 if (err)
472 {
473 HIBLOG("IOPolledInterface::checkForWork[%d] 0x%x\n", idx, err);
474 }
475 else
476 {
477 err = vars->ioStatus;
478 if (kIOReturnSuccess != err)
479 HIBLOG("IOPolledInterface::ioStatus 0x%x\n", err);
480 }
3a60a9f5 481
2d21ac55 482 return (err);
3a60a9f5
A		 483}
484
485IOReturn
486IOPolledInterface::checkAllForWork(void)
487{
488 IOReturn err = kIOReturnNotReady;
489 int32_t idx;
490 IOPolledInterface * poller;
491
492 IOHibernateVars * vars = &gIOHibernateVars;
493
494 if (!vars->fileVars || !vars->fileVars->pollers)
495 return (err);
496
497 for (idx = 0;
498 (poller = (IOPolledInterface *) vars->fileVars->pollers->getObject(idx));
499 idx++)
500 {
501 err = poller->checkForWork();
502 if (err)
503 HIBLOG("IOPolledInterface::checkAllForWork[%d] 0x%x\n", idx, err);
504 }
505
506 return (err);
507}
508
509/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
510
511struct _OpenFileContext
512{
513 OSData * extents;
514 uint64_t size;
515};
516
517static void
518file_extent_callback(void * ref, uint64_t start, uint64_t length)
519{
520 _OpenFileContext * ctx = (_OpenFileContext *) ref;
521 IOPolledFileExtent extent;
522
523 extent.start = start;
524 extent.length = length;
525
526 ctx->extents->appendBytes(&extent, sizeof(extent));
527 ctx->size += length;
528}
529
530IOReturn
531IOPolledFileOpen( const char * filename, IOBufferMemoryDescriptor * ioBuffer,
532 IOPolledFileIOVars ** fileVars, OSData ** fileExtents,
533 OSData ** imagePath)
534{
535 IOReturn err = kIOReturnError;
536 IOPolledFileIOVars * vars;
537 _OpenFileContext ctx;
538 OSData * extentsData;
539 OSNumber * num;
540 IORegistryEntry * part = 0;
541 OSDictionary * matching;
542 OSIterator * iter;
543 dev_t hibernate_image_dev;
544 uint64_t maxiobytes;
545
546 vars = &gFileVars;
547 do
548 {
549 HIBLOG("sizeof(IOHibernateImageHeader) == %ld\n", sizeof(IOHibernateImageHeader));
550 if (sizeof(IOHibernateImageHeader) != 512)
551 continue;
552
553 vars->io = false;
554 vars->buffer = (uint8_t *) ioBuffer->getBytesNoCopy();
555 vars->bufferHalf = 0;
556 vars->bufferOffset = 0;
557 vars->bufferSize = ioBuffer->getLength() >> 1;
558
559 extentsData = OSData::withCapacity(32);
560
561 ctx.extents = extentsData;
562 ctx.size = 0;
563 vars->fileRef = kern_open_file_for_direct_io(filename,
564 &file_extent_callback, &ctx,
565 &hibernate_image_dev,
566 &vars->block0,
567 &maxiobytes);
568 if (!vars->fileRef)
569 {
570 err = kIOReturnNoSpace;
571 break;
572 }
b0d623f7 573 gIOHibernateFileRef = vars->fileRef;
3a60a9f5
A		 574 HIBLOG("Opened file %s, size %qd, partition base 0x%qx, maxio %qx\n", filename, ctx.size,
575 vars->block0, maxiobytes);
576 if (ctx.size < 1*1024*1024) // check against image size estimate!
577 {
578 err = kIOReturnNoSpace;
579 break;
580 }
581
582 if (maxiobytes < vars->bufferSize)
583 vars->bufferSize = maxiobytes;
584
585 vars->extentMap = (IOPolledFileExtent *) extentsData->getBytesNoCopy();
586
587 matching = IOService::serviceMatching("IOMedia");
588 num = OSNumber::withNumber(major(hibernate_image_dev), 32);
589 matching->setObject(kIOBSDMajorKey, num);
590 num->release();
591 num = OSNumber::withNumber(minor(hibernate_image_dev), 32);
592 matching->setObject(kIOBSDMinorKey, num);
593 num->release();
594 iter = IOService::getMatchingServices(matching);
595 matching->release();
596 if (iter)
597 {
598 part = (IORegistryEntry *) iter->getNextObject();
599 part->retain();
600 iter->release();
601 }
2d21ac55
A		 602 if (!part)
603 break;
604
3a60a9f5
A		 605 int minor, major;
606 IORegistryEntry * next;
607 IORegistryEntry * child;
608 OSData * data;
609
610 num = (OSNumber *) part->getProperty(kIOBSDMajorKey);
611 if (!num)
612 break;
613 major = num->unsigned32BitValue();
614 num = (OSNumber *) part->getProperty(kIOBSDMinorKey);
615 if (!num)
616 break;
617 minor = num->unsigned32BitValue();
618
619 hibernate_image_dev = makedev(major, minor);
620
621 vars->pollers = OSArray::withCapacity(4);
622 if (!vars->pollers)
623 break;
624
625 vars->blockSize = 512;
626 next = part;
627 do
628 {
629 IOPolledInterface * poller;
c0fea474
A		 630 OSObject * obj;
631
632 obj = next->getProperty(kIOPolledInterfaceSupportKey);
633 if (kOSBooleanFalse == obj)
634 {
635 vars->pollers->flushCollection();
636 break;
637 }
638 else if ((poller = OSDynamicCast(IOPolledInterface, obj)))
3a60a9f5
A		 639 vars->pollers->setObject(poller);
640 if ((num = OSDynamicCast(OSNumber, next->getProperty(kIOMediaPreferredBlockSizeKey))))
641 vars->blockSize = num->unsigned32BitValue();
642 child = next;
643 }
644 while ((next = child->getParentEntry(gIOServicePlane))
645 && child->isParent(next, gIOServicePlane, true));
646
647 HIBLOG("hibernate image major %d, minor %d, blocksize %ld, pollers %d\n",
b0d623f7 648 major, minor, (long)vars->blockSize, vars->pollers->getCount());
3a60a9f5
A		 649 if (vars->pollers->getCount() < kIOHibernateMinPollersNeeded)
650 continue;
651
652 err = IOHibernatePollerProbe(vars, (IOService *) part);
653 if (kIOReturnSuccess != err)
654 break;
655
656 err = IOHibernatePollerOpen(vars, kIOPolledPreflightState, ioBuffer);
657 if (kIOReturnSuccess != err)
658 break;
659
660 *fileVars = vars;
661 *fileExtents = extentsData;
662
663 // make imagePath
3a60a9f5 664
0c530ab8 665 if ((extentsData->getLength() >= sizeof(IOPolledFileExtent)))
3a60a9f5 666 {
0c530ab8
A		 667 char str2[24];
668
b0d623f7 669#if defined(__i386__) || defined(__x86_64__)
0c530ab8
A		 670 if (!gIOCreateEFIDevicePathSymbol)
671 gIOCreateEFIDevicePathSymbol = OSSymbol::withCString("CreateEFIDevicePath");
672
2d21ac55 673 snprintf(str2, sizeof(str2), "%qx", vars->extentMap[0].start);
0c530ab8
A		 674
675 err = IOService::getPlatform()->callPlatformFunction(
676 gIOCreateEFIDevicePathSymbol, false,
677 (void *) part, (void *) str2, (void *) true,
678 (void *) &data);
679#else
680 char str1[256];
681 int len = sizeof(str1);
682
683 if (!part->getPath(str1, &len, gIODTPlane))
684 err = kIOReturnNotFound;
685 else
686 {
2d21ac55 687 snprintf(str2, sizeof(str2), ",%qx", vars->extentMap[0].start);
0c530ab8
A		 688 // (strip the plane name)
689 char * tail = strchr(str1, ':');
690 if (!tail)
691 tail = str1 - 1;
692 data = OSData::withBytes(tail + 1, strlen(tail + 1));
693 data->appendBytes(str2, strlen(str2));
694 }
695#endif
696 if (kIOReturnSuccess == err)
3a60a9f5 697 *imagePath = data;
0c530ab8
A		 698 else
699 HIBLOG("error 0x%x getting path\n", err);
3a60a9f5
A		 700 }
701 }
702 while (false);
703
704 if (kIOReturnSuccess != err)
705 {
706 HIBLOG("error 0x%x opening hibernation file\n", err);
707 if (vars->fileRef)
b0d623f7 708 {
3a60a9f5 709 kern_close_file_for_direct_io(vars->fileRef);
b0d623f7
A		 710 gIOHibernateFileRef = vars->fileRef = NULL;
711 }
3a60a9f5
A		 712 }
713
714 if (part)
715 part->release();
716
717 return (err);
718}
719
720IOReturn
721IOPolledFileClose( IOPolledFileIOVars * vars )
722{
723 if (vars->pollers)
724 {
725 IOHibernatePollerClose(vars, kIOPolledPostflightState);
726 vars->pollers->release();
727 }
728
3a60a9f5
A		 729 bzero(vars, sizeof(IOPolledFileIOVars));
730
731 return (kIOReturnSuccess);
732}
733
734static IOReturn
735IOPolledFileSeek(IOPolledFileIOVars * vars, uint64_t position)
736{
737 IOPolledFileExtent * extentMap;
738
739 extentMap = vars->extentMap;
740
741 vars->position = position;
742
743 while (position >= extentMap->length)
744 {
745 position -= extentMap->length;
746 extentMap++;
747 }
748
749 vars->currentExtent = extentMap;
750 vars->extentRemaining = extentMap->length - position;
751 vars->extentPosition = vars->position - position;
752
753 if (vars->bufferSize <= vars->extentRemaining)
754 vars->bufferLimit = vars->bufferSize;
755 else
756 vars->bufferLimit = vars->extentRemaining;
757
758 return (kIOReturnSuccess);
759}
760
761static IOReturn
762IOPolledFileWrite(IOPolledFileIOVars * vars,
763 const uint8_t * bytes, IOByteCount size,
764 hibernate_cryptvars_t * cryptvars)
765{
766 IOReturn err = kIOReturnSuccess;
767 IOByteCount copy;
768 bool flush = false;
769
770 do
771 {
772 if (!bytes && !size)
773 {
774 // seek to end of block & flush
775 size = vars->position & (vars->blockSize - 1);
776 if (size)
777 size = vars->blockSize - size;
778 flush = true;
779 // use some garbage for the fill
780 bytes = vars->buffer + vars->bufferOffset;
781 }
782
783 copy = vars->bufferLimit - vars->bufferOffset;
784 if (copy > size)
785 copy = size;
786 else
787 flush = true;
788
789 if (bytes)
790 {
791 bcopy(bytes, vars->buffer + vars->bufferHalf + vars->bufferOffset, copy);
792 bytes += copy;
793 }
794 else
795 bzero(vars->buffer + vars->bufferHalf + vars->bufferOffset, copy);
796
797 size -= copy;
798 vars->bufferOffset += copy;
799 vars->position += copy;
800
801 if (flush && vars->bufferOffset)
802 {
803 uint64_t offset = (vars->position - vars->bufferOffset
804 - vars->extentPosition + vars->currentExtent->start);
805 uint32_t length = (vars->bufferOffset);
806
2d21ac55 807#if CRYPTO
3a60a9f5
A		 808 if (cryptvars && vars->encryptStart && (vars->position > vars->encryptStart))
809 {
810 uint32_t encryptLen, encryptStart;
811 encryptLen = vars->position - vars->encryptStart;
812 if (encryptLen > length)
813 encryptLen = length;
814 encryptStart = length - encryptLen;
815
816 // encrypt the buffer
817 aes_encrypt_cbc(vars->buffer + vars->bufferHalf + encryptStart,
818 &cryptvars->aes_iv[0],
819 encryptLen / AES_BLOCK_SIZE,
820 vars->buffer + vars->bufferHalf + encryptStart,
821 &cryptvars->ctx.encrypt);
822 // save initial vector for following encrypts
823 bcopy(vars->buffer + vars->bufferHalf + encryptStart + encryptLen - AES_BLOCK_SIZE,
824 &cryptvars->aes_iv[0],
825 AES_BLOCK_SIZE);
826 }
2d21ac55 827#endif /* CRYPTO */
3a60a9f5
A		 828
829 if (vars->io)
830 {
2d21ac55 831 err = IOHibernatePollerIODone(vars, true);
3a60a9f5
A		 832 if (kIOReturnSuccess != err)
833 break;
834 }
835
836if (vars->position & (vars->blockSize - 1)) HIBLOG("misaligned file pos %qx\n", vars->position);
837//if (length != vars->bufferSize) HIBLOG("short write of %qx ends@ %qx\n", length, offset + length);
838
839 err = IOHibernatePollerIO(vars, kIOPolledWrite, vars->bufferHalf, offset, length);
840 if (kIOReturnSuccess != err)
841 break;
842 vars->io = true;
843
844 vars->extentRemaining -= vars->bufferOffset;
845 if (!vars->extentRemaining)
846 {
847 vars->currentExtent++;
848 vars->extentRemaining = vars->currentExtent->length;
849 vars->extentPosition = vars->position;
850 if (!vars->extentRemaining)
851 {
852 err = kIOReturnOverrun;
853 break;
854 }
855 }
856
857 vars->bufferHalf = vars->bufferHalf ? 0 : vars->bufferSize;
858 vars->bufferOffset = 0;
859 if (vars->bufferSize <= vars->extentRemaining)
860 vars->bufferLimit = vars->bufferSize;
861 else
862 vars->bufferLimit = vars->extentRemaining;
863
864 flush = false;
865 }
866 }
867 while (size);
868
869 return (err);
870}
871
872static IOReturn
873IOPolledFileRead(IOPolledFileIOVars * vars,
874 uint8_t * bytes, IOByteCount size,
875 hibernate_cryptvars_t * cryptvars)
876{
877 IOReturn err = kIOReturnSuccess;
878 IOByteCount copy;
879
880// bytesWritten += size;
881
882 do
883 {
884 copy = vars->bufferLimit - vars->bufferOffset;
885 if (copy > size)
886 copy = size;
887
888 if (bytes)
889 {
890 bcopy(vars->buffer + vars->bufferHalf + vars->bufferOffset, bytes, copy);
891 bytes += copy;
892 }
893 size -= copy;
894 vars->bufferOffset += copy;
895// vars->position += copy;
896
897 if (vars->bufferOffset == vars->bufferLimit)
898 {
899 if (vars->io)
900 {
2d21ac55 901 err = IOHibernatePollerIODone(vars, false);
3a60a9f5
A		 902 if (kIOReturnSuccess != err)
903 break;
904 }
905 else
906 cryptvars = 0;
907
908if (vars->position & (vars->blockSize - 1)) HIBLOG("misaligned file pos %qx\n", vars->position);
909
910 vars->position += vars->lastRead;
911 vars->extentRemaining -= vars->lastRead;
912 vars->bufferLimit = vars->lastRead;
913
914 if (!vars->extentRemaining)
915 {
916 vars->currentExtent++;
917 vars->extentRemaining = vars->currentExtent->length;
918 vars->extentPosition = vars->position;
919 if (!vars->extentRemaining)
920 {
921 err = kIOReturnOverrun;
922 break;
923 }
924 }
925
36401178
A		 926 uint64_t length;
927 uint64_t lastReadLength = vars->lastRead;
3a60a9f5
A		 928 uint64_t offset = (vars->position
929 - vars->extentPosition + vars->currentExtent->start);
36401178
A		 930 if (vars->extentRemaining <= vars->bufferSize)
931 length = vars->extentRemaining;
932 else
933 length = vars->bufferSize;
934 vars->lastRead = length;
3a60a9f5
A		 935
936//if (length != vars->bufferSize) HIBLOG("short read of %qx ends@ %qx\n", length, offset + length);
937
938 err = IOHibernatePollerIO(vars, kIOPolledRead, vars->bufferHalf, offset, length);
939 if (kIOReturnSuccess != err)
940 break;
941 vars->io = true;
942
943 vars->bufferHalf = vars->bufferHalf ? 0 : vars->bufferSize;
944 vars->bufferOffset = 0;
945
2d21ac55 946#if CRYPTO
3a60a9f5
A		 947 if (cryptvars)
948 {
949 uint8_t thisVector[AES_BLOCK_SIZE];
950 // save initial vector for following decrypts
951 bcopy(&cryptvars->aes_iv[0], &thisVector[0], AES_BLOCK_SIZE);
36401178 952 bcopy(vars->buffer + vars->bufferHalf + lastReadLength - AES_BLOCK_SIZE,
3a60a9f5
A		 953 &cryptvars->aes_iv[0], AES_BLOCK_SIZE);
954 // decrypt the buffer
955 aes_decrypt_cbc(vars->buffer + vars->bufferHalf,
956 &thisVector[0],
36401178 957 lastReadLength / AES_BLOCK_SIZE,
3a60a9f5
A		 958 vars->buffer + vars->bufferHalf,
959 &cryptvars->ctx.decrypt);
960 }
b0d623f7 961#endif /* CRYPTO */
3a60a9f5
A		 962 }
963 }
964 while (size);
965
966 return (err);
967}
968
969/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
970
971IOReturn
972IOHibernateSystemSleep(void)
973{
974 IOReturn err;
975 OSData * data;
976 OSObject * obj;
977 OSString * str;
978 OSNumber * num;
2d21ac55 979 OSDictionary *sleepOverrideOptions;
3a60a9f5
A		 980
981 IOHibernateVars * vars = &gIOHibernateVars;
982
983 if (vars->fileVars && vars->fileVars->fileRef)
984 // already on the way down
985 return (kIOReturnSuccess);
986
987 gIOHibernateState = kIOHibernateStateInactive;
988
b0d623f7
A		 989 gIOHibernateDebugFlags = 0;
990 if (kIOLogHibernate & gIOKitDebug)
991 gIOHibernateDebugFlags |= kIOHibernateDebugRestoreLogs;
992
2d21ac55
A		 993 /* The invocation of IOPMSleepSystemWithOptions() may override
994 * existing hibernation settings.
995 */
996 sleepOverrideOptions = (OSDictionary *)OSDynamicCast( OSDictionary,
997 IOService::getPMRootDomain()->copyProperty(kRootDomainSleepOptionsKey));
998
999
1000 /* Hibernate mode overriden by sleep otions ? */
1001 obj = NULL;
1002
1003 if (sleepOverrideOptions) {
1004 obj = sleepOverrideOptions->getObject(kIOHibernateModeKey);
1005 if (obj) obj->retain();
1006 }
1007
1008 if(!obj) {
1009 obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateModeKey);
1010 }
1011
1012 if (obj && (num = OSDynamicCast(OSNumber, obj)) )
3a60a9f5 1013 {
3a60a9f5
A		 1014 gIOHibernateMode = num->unsigned32BitValue();
1015 if (kIOHibernateModeSleep & gIOHibernateMode)
1016 // default to discard clean for safe sleep
1017 gIOHibernateMode ^= (kIOHibernateModeDiscardCleanInactive
1018 | kIOHibernateModeDiscardCleanActive);
2d21ac55 1019 }
3a60a9f5 1020
2d21ac55
A		 1021 if (obj) obj->release();
1022
1023 /* Hibernate free rotio overriden by sleep options ? */
1024 obj = NULL;
1025
1026 if (sleepOverrideOptions) {
1027 obj = sleepOverrideOptions->getObject(kIOHibernateFreeRatioKey);
1028 if (obj) obj->retain();
3a60a9f5 1029 }
2d21ac55
A		 1030
1031 if(!obj) {
1032 obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateFreeRatioKey);
1033 }
1034 if (obj && (num = OSDynamicCast(OSNumber, obj)))
3a60a9f5 1035 {
3a60a9f5 1036 gIOHibernateFreeRatio = num->unsigned32BitValue();
3a60a9f5 1037 }
2d21ac55
A		 1038 if (obj) obj->release();
1039
3a60a9f5
A		 1040 if ((obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateFreeTimeKey)))
1041 {
1042 if ((num = OSDynamicCast(OSNumber, obj)))
1043 gIOHibernateFreeTime = num->unsigned32BitValue();
1044 obj->release();
1045 }
1046 if ((obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateFileKey)))
1047 {
1048 if ((str = OSDynamicCast(OSString, obj)))
2d21ac55
A		 1049 strlcpy(gIOHibernateFilename, str->getCStringNoCopy(),
1050 sizeof(gIOHibernateFilename));
3a60a9f5
A		 1051 obj->release();
1052 }
1053
2d21ac55
A		 1054 if (sleepOverrideOptions)
1055 sleepOverrideOptions->release();
1056
3a60a9f5
A		 1057 if (!gIOHibernateMode || !gIOHibernateFilename[0])
1058 return (kIOReturnUnsupported);
1059
1060 HIBLOG("hibernate image path: %s\n", gIOHibernateFilename);
1061
2d21ac55 1062
3a60a9f5
A		 1063 do
1064 {
0c530ab8
A		 1065 vars->srcBuffer = IOBufferMemoryDescriptor::withOptions(kIODirectionOutIn,
1066 4 * page_size, page_size);
1067 vars->ioBuffer = IOBufferMemoryDescriptor::withOptions(kIODirectionOutIn,
1068 2 * kDefaultIOSize, page_size);
3a60a9f5
A		 1069
1070 if (!vars->srcBuffer || !vars->ioBuffer)
1071 {
1072 err = kIOReturnNoMemory;
1073 break;
1074 }
1075
1076 err = IOPolledFileOpen(gIOHibernateFilename, vars->ioBuffer,
1077 &vars->fileVars, &vars->fileExtents, &data);
1078 if (KERN_SUCCESS != err)
1079 {
1080 HIBLOG("IOPolledFileOpen(%x)\n", err);
1081 break;
1082 }
1083 if (vars->fileVars->fileRef)
1084 {
1085 // invalidate the image file
1086 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderInvalidSignature;
1087 int err = kern_write_file(vars->fileVars->fileRef, 0,
1088 (caddr_t) gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
1089 if (KERN_SUCCESS != err)
1090 HIBLOG("kern_write_file(%d)\n", err);
1091 }
1092
1093 bzero(gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
b0d623f7 1094 gIOHibernateCurrentHeader->debugFlags = gIOHibernateDebugFlags;
3a60a9f5
A		 1095
1096 boolean_t encryptedswap;
1097 err = hibernate_setup(gIOHibernateCurrentHeader,
1098 gIOHibernateFreeRatio, gIOHibernateFreeTime,
1099 &vars->page_list, &vars->page_list_wired, &encryptedswap);
1100 if (KERN_SUCCESS != err)
1101 {
1102 HIBLOG("hibernate_setup(%d)\n", err);
1103 break;
1104 }
1105
1106 if (encryptedswap)
1107 gIOHibernateMode ^= kIOHibernateModeEncrypt;
1108
1109 vars->videoAllocSize = kVideoMapSize;
1110 if (KERN_SUCCESS != kmem_alloc_pageable(kernel_map, &vars->videoMapping, vars->videoAllocSize))
1111 vars->videoMapping = 0;
1112
1113 // generate crypt keys
1114 for (uint32_t i = 0; i < sizeof(vars->wiredCryptKey); i++)
1115 vars->wiredCryptKey[i] = random();
1116 for (uint32_t i = 0; i < sizeof(vars->cryptKey); i++)
1117 vars->cryptKey[i] = random();
1118
1119 // set nvram
1120
1121 IORegistryEntry * regEntry;
1122 if (!gIOOptionsEntry)
1123 {
1124 regEntry = IORegistryEntry::fromPath("/options", gIODTPlane);
1125 gIOOptionsEntry = OSDynamicCast(IODTNVRAM, regEntry);
1126 if (regEntry && !gIOOptionsEntry)
1127 regEntry->release();
1128 }
1129 if (!gIOChosenEntry)
1130 gIOChosenEntry = IORegistryEntry::fromPath("/chosen", gIODTPlane);
1131
1132 if (gIOOptionsEntry)
1133 {
1134 const OSSymbol * sym;
3a60a9f5
A		 1135
1136 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKey);
1137 if (sym)
1138 {
1139 gIOOptionsEntry->setProperty(sym, data);
1140 sym->release();
1141 }
1142 data->release();
1143
b0d623f7 1144#if defined(__ppc__)
0c530ab8
A		 1145 size_t len;
1146 char valueString[16];
1147
3a60a9f5
A		 1148 vars->saveBootDevice = gIOOptionsEntry->copyProperty(kIOSelectedBootDeviceKey);
1149 if (gIOChosenEntry)
1150 {
1151 OSData * bootDevice = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOBootPathKey));
1152 if (bootDevice)
1153 {
1154 sym = OSSymbol::withCStringNoCopy(kIOSelectedBootDeviceKey);
1155 OSString * str2 = OSString::withCStringNoCopy((const char *) bootDevice->getBytesNoCopy());
1156 if (sym && str2)
1157 gIOOptionsEntry->setProperty(sym, str2);
1158 if (sym)
1159 sym->release();
1160 if (str2)
1161 str2->release();
1162 }
3a60a9f5
A		 1163 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMemorySignatureKey));
1164 if (data)
1165 {
1166 vars->haveFastBoot = true;
1167
b0d623f7 1168 len = snprintf(valueString, sizeof(valueString), "0x%lx", *((UInt32 *)data->getBytesNoCopy()));
3a60a9f5
A		 1169 data = OSData::withBytes(valueString, len + 1);
1170 sym = OSSymbol::withCStringNoCopy(kIOHibernateMemorySignatureEnvKey);
1171 if (sym && data)
1172 gIOOptionsEntry->setProperty(sym, data);
1173 if (sym)
1174 sym->release();
1175 if (data)
1176 data->release();
1177 }
1178 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMachineSignatureKey));
1179 if (data)
1180 gIOHibernateCurrentHeader->machineSignature = *((UInt32 *)data->getBytesNoCopy());
1181 }
0c530ab8 1182#endif /* __ppc__ */
b0d623f7 1183#if defined(__i386__) || defined(__x86_64__)
0c530ab8
A		 1184 struct AppleRTCHibernateVars
1185 {
1186 uint8_t signature[4];
1187 uint32_t revision;
1188 uint8_t booterSignature[20];
1189 uint8_t wiredCryptKey[16];
1190 };
1191 AppleRTCHibernateVars rtcVars;
1192
1193 rtcVars.signature[0] = 'A';
1194 rtcVars.signature[1] = 'A';
1195 rtcVars.signature[2] = 'P';
1196 rtcVars.signature[3] = 'L';
1197 rtcVars.revision = 1;
1198 bcopy(&vars->wiredCryptKey[0], &rtcVars.wiredCryptKey[0], sizeof(rtcVars.wiredCryptKey));
1199 if (gIOHibernateBootSignature[0])
1200 {
1201 char c;
1202 uint8_t value = 0;
1203 for (uint32_t i = 0;
1204 (c = gIOHibernateBootSignature[i]) && (i < (sizeof(rtcVars.booterSignature) << 1));
1205 i++)
1206 {
1207 if (c >= 'a')
1208 c -= 'a' - 10;
1209 else if (c >= 'A')
1210 c -= 'A' - 10;
1211 else if (c >= '0')
1212 c -= '0';
1213 else
1214 continue;
1215 value = (value << 4) | c;
1216 if (i & 1)
1217 rtcVars.booterSignature[i >> 1] = value;
1218 }
1219 }
1220 data = OSData::withBytes(&rtcVars, sizeof(rtcVars));
1221 if (data)
2d21ac55
A		 1222 {
1223 IOService::getPMRootDomain()->setProperty(kIOHibernateRTCVariablesKey, data);
1224
1225 if( gIOOptionsEntry )
1226 {
1227 if( gIOHibernateMode & kIOHibernateModeSwitch )
1228 {
1229 const OSSymbol *sym;
1230 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootSwitchVarsKey);
1231 if( sym )
1232 {
1233 gIOOptionsEntry->setProperty(sym, data); /* intentional insecure backup of rtc boot vars */
1234 sym->release();
1235 }
1236 }
1237 }
1238
1239 data->release();
0c530ab8
A		 1240 }
1241 if (gIOChosenEntry)
1242 {
1243 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMachineSignatureKey));
1244 if (data)
1245 gIOHibernateCurrentHeader->machineSignature = *((UInt32 *)data->getBytesNoCopy());
1246 }
b0d623f7 1247#else /* !i386 && !x86_64 */
3a60a9f5
A		 1248 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1249 {
1250 data = OSData::withBytes(&vars->wiredCryptKey[0], sizeof(vars->wiredCryptKey));
1251 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKeyKey);
1252 if (sym && data)
1253 gIOOptionsEntry->setProperty(sym, data);
1254 if (sym)
1255 sym->release();
1256 if (data)
1257 data->release();
2d21ac55 1258 if (false && gIOHibernateBootSignature[0])
3a60a9f5
A		 1259 {
1260 data = OSData::withCapacity(16);
1261 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootSignatureKey);
1262 if (sym && data)
1263 {
1264 char c;
0c530ab8 1265 uint8_t value = 0;
3a60a9f5
A		 1266 for (uint32_t i = 0; (c = gIOHibernateBootSignature[i]); i++)
1267 {
1268 if (c >= 'a')
1269 c -= 'a' - 10;
1270 else if (c >= 'A')
1271 c -= 'A' - 10;
1272 else if (c >= '0')
1273 c -= '0';
1274 else
1275 continue;
1276 value = (value << 4) | c;
1277 if (i & 1)
1278 data->appendBytes(&value, sizeof(value));
1279 }
1280 gIOOptionsEntry->setProperty(sym, data);
1281 }
1282 if (sym)
1283 sym->release();
1284 if (data)
1285 data->release();
1286 }
1287 }
3a60a9f5
A		 1288 if (!vars->haveFastBoot)
1289 {
1290 // set boot volume to zero
1291 IODTPlatformExpert * platform = OSDynamicCast(IODTPlatformExpert, IOService::getPlatform());
1292 if (platform && (kIOReturnSuccess == platform->readXPRAM(kXPRamAudioVolume,
1293 &vars->saveBootAudioVolume, sizeof(vars->saveBootAudioVolume))))
1294 {
1295 uint8_t newVolume;
1296 newVolume = vars->saveBootAudioVolume & 0xf8;
1297 platform->writeXPRAM(kXPRamAudioVolume,
1298 &newVolume, sizeof(newVolume));
1299 }
1300 }
b0d623f7 1301#endif /* !i386 && !x86_64 */
3a60a9f5
A		 1302 }
1303 // --
1304
1305 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderSignature;
1306 gIOHibernateState = kIOHibernateStateHibernating;
1307 }
1308 while (false);
1309
1310 return (err);
1311}
1312
1313/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1314
2d21ac55
A		 1315DECLARE_IOHIBERNATEPROGRESSALPHA
1316
1317static void
1318ProgressInit(hibernate_graphics_t * display, uint8_t * screen, uint8_t * saveunder, uint32_t savelen)
1319{
1320 uint32_t rowBytes, pixelShift;
1321 uint32_t x, y;
1322 int32_t blob;
1323 uint32_t alpha, in, color, result;
1324 uint8_t * out;
1325 uint32_t saveindex[kIOHibernateProgressCount] = { 0 };
1326
1327 rowBytes = display->rowBytes;
1328 pixelShift = display->depth >> 4;
1329 if (pixelShift < 1) return;
1330
1331 screen += ((display->width
1332 - kIOHibernateProgressCount * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << (pixelShift - 1))
1333 + (display->height - kIOHibernateProgressOriginY - kIOHibernateProgressHeight) * rowBytes;
1334
1335 for (y = 0; y < kIOHibernateProgressHeight; y++)
1336 {
1337 out = screen + y * rowBytes;
1338 for (blob = 0; blob < kIOHibernateProgressCount; blob++)
1339 {
1340 color = blob ? kIOHibernateProgressDarkGray : kIOHibernateProgressMidGray;
1341 for (x = 0; x < kIOHibernateProgressWidth; x++)
1342 {
1343 alpha = gIOHibernateProgressAlpha[y][x];
1344 result = color;
1345 if (alpha)
1346 {
1347 if (0xff != alpha)
1348 {
1349 if (1 == pixelShift)
1350 {
1351 in = *((uint16_t *)out) & 0x1f; // 16
1352 in = (in << 3) | (in >> 2);
1353 }
1354 else
1355 in = *((uint32_t *)out) & 0xff; // 32
1356 saveunder[blob * kIOHibernateProgressSaveUnderSize + saveindex[blob]++] = in;
1357 result = ((255 - alpha) * in + alpha * result + 0xff) >> 8;
1358 }
1359 if (1 == pixelShift)
1360 {
1361 result >>= 3;
1362 *((uint16_t *)out) = (result << 10) | (result << 5) | result; // 16
1363 }
1364 else
1365 *((uint32_t *)out) = (result << 16) | (result << 8) | result; // 32
1366 }
1367 out += (1 << pixelShift);
1368 }
1369 out += (kIOHibernateProgressSpacing << pixelShift);
1370 }
1371 }
1372}
1373
1374
1375static void
1376ProgressUpdate(hibernate_graphics_t * display, uint8_t * screen, int32_t firstBlob, int32_t select)
1377{
1378 uint32_t rowBytes, pixelShift;
1379 uint32_t x, y;
1380 int32_t blob, lastBlob;
1381 uint32_t alpha, in, color, result;
1382 uint8_t * out;
1383 uint32_t saveindex[kIOHibernateProgressCount] = { 0 };
1384
1385 pixelShift = display->depth >> 4;
1386 if (pixelShift < 1)
1387 return;
1388
1389 rowBytes = display->rowBytes;
1390
1391 screen += ((display->width
1392 - kIOHibernateProgressCount * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << (pixelShift - 1))
1393 + (display->height - kIOHibernateProgressOriginY - kIOHibernateProgressHeight) * rowBytes;
1394
1395 lastBlob = (select < kIOHibernateProgressCount) ? select : (kIOHibernateProgressCount - 1);
1396
1397 screen += (firstBlob * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << pixelShift;
1398
1399 for (y = 0; y < kIOHibernateProgressHeight; y++)
1400 {
1401 out = screen + y * rowBytes;
1402 for (blob = firstBlob; blob <= lastBlob; blob++)
1403 {
1404 color = (blob < select) ? kIOHibernateProgressLightGray : kIOHibernateProgressMidGray;
1405 for (x = 0; x < kIOHibernateProgressWidth; x++)
1406 {
1407 alpha = gIOHibernateProgressAlpha[y][x];
1408 result = color;
1409 if (alpha)
1410 {
1411 if (0xff != alpha)
1412 {
1413 in = display->progressSaveUnder[blob][saveindex[blob]++];
1414 result = ((255 - alpha) * in + alpha * result + 0xff) / 255;
1415 }
1416 if (1 == pixelShift)
1417 {
1418 result >>= 3;
1419 *((uint16_t *)out) = (result << 10) | (result << 5) | result; // 16
1420 }
1421 else
1422 *((uint32_t *)out) = (result << 16) | (result << 8) | result; // 32
1423 }
1424 out += (1 << pixelShift);
1425 }
1426 out += (kIOHibernateProgressSpacing << pixelShift);
1427 }
1428 }
1429}
1430
1431/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1432
3a60a9f5
A		 1433IOReturn
1434IOHibernateSystemHasSlept(void)
1435{
1436 IOHibernateVars * vars = &gIOHibernateVars;
2d21ac55
A		 1437 OSObject * obj;
1438 OSData * data;
1439
1440 obj = IOService::getPMRootDomain()->copyProperty(kIOHibernatePreviewBufferKey);
1441 vars->previewBuffer = OSDynamicCast(IOMemoryDescriptor, obj);
1442 if (obj && !vars->previewBuffer)
1443 obj->release();
3a60a9f5 1444
2d21ac55
A		 1445 vars->consoleMapping = NULL;
1446 if (vars->previewBuffer && (kIOReturnSuccess != vars->previewBuffer->prepare()))
3a60a9f5 1447 {
2d21ac55
A		 1448 vars->previewBuffer->release();
1449 vars->previewBuffer = 0;
1450 }
3a60a9f5 1451
2d21ac55
A		 1452 if (vars->previewBuffer && (data = OSDynamicCast(OSData,
1453 IOService::getPMRootDomain()->getProperty(kIOHibernatePreviewActiveKey))))
1454 {
1455 UInt32 flags = *((UInt32 *)data->getBytesNoCopy());
b0d623f7 1456 HIBPRINT("kIOHibernatePreviewActiveKey %08lx\n", (long)flags);
2d21ac55
A		 1457
1458 IOService::getPMRootDomain()->removeProperty(kIOHibernatePreviewActiveKey);
1459
1460 if (kIOHibernatePreviewUpdates & flags)
1461 {
1462 PE_Video consoleInfo;
1463 hibernate_graphics_t * graphicsInfo = gIOHibernateGraphicsInfo;
1464
1465 IOService::getPlatform()->getConsoleInfo(&consoleInfo);
1466
1467 graphicsInfo->width = consoleInfo.v_width;
1468 graphicsInfo->height = consoleInfo.v_height;
1469 graphicsInfo->rowBytes = consoleInfo.v_rowBytes;
1470 graphicsInfo->depth = consoleInfo.v_depth;
1471 vars->consoleMapping = (uint8_t *) consoleInfo.v_baseAddr;
1472
1473 HIBPRINT("video %p %d %d %d\n",
1474 vars->consoleMapping, gIOHibernateGraphicsInfo->depth,
1475 gIOHibernateGraphicsInfo->width, gIOHibernateGraphicsInfo->height);
1476 if (vars->consoleMapping)
1477 ProgressInit(graphicsInfo, vars->consoleMapping,
1478 &graphicsInfo->progressSaveUnder[0][0], sizeof(graphicsInfo->progressSaveUnder));
1479 }
3a60a9f5 1480 }
2d21ac55 1481
3a60a9f5
A		 1482 if (gIOOptionsEntry)
1483 gIOOptionsEntry->sync();
1484
1485 return (kIOReturnSuccess);
1486}
1487
1488/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1489
1490IOReturn
1491IOHibernateSystemWake(void)
1492{
1493 IOHibernateVars * vars = &gIOHibernateVars;
1494
1495 hibernate_teardown(vars->page_list, vars->page_list_wired);
1496
1497 if (vars->videoMapping)
1498 {
1499 if (vars->videoMapSize)
1500 // remove mappings
1501 IOUnmapPages(kernel_map, vars->videoMapping, vars->videoMapSize);
1502 if (vars->videoAllocSize)
1503 // dealloc range
b0d623f7 1504 kmem_free(kernel_map, trunc_page(vars->videoMapping), vars->videoAllocSize);
3a60a9f5
A		 1505 }
1506
1507 if (vars->previewBuffer)
1508 {
1509 vars->previewBuffer->release();
1510 vars->previewBuffer = 0;
1511 }
1512
1513 if (vars->fileVars)
1514 {
1515 IOPolledFileClose(vars->fileVars);
1516 }
1517
1518 // invalidate nvram properties - (gIOOptionsEntry != 0) => nvram was touched
1519
0c530ab8 1520#ifdef __ppc__
3a60a9f5
A		 1521 OSData * data = OSData::withCapacity(4);
1522 if (gIOOptionsEntry && data)
1523 {
1524 const OSSymbol * sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKey);
1525 if (sym)
1526 {
1527 gIOOptionsEntry->setProperty(sym, data);
1528 sym->release();
1529 }
1530 sym = OSSymbol::withCStringNoCopy(kIOSelectedBootDeviceKey);
1531 if (sym)
1532 {
1533 if (vars->saveBootDevice)
1534 {
1535 gIOOptionsEntry->setProperty(sym, vars->saveBootDevice);
1536 vars->saveBootDevice->release();
1537 }
1538 sym->release();
1539 }
1540 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKeyKey);
1541 if (sym)
1542 {
1543 gIOOptionsEntry->setProperty(sym, data);
1544 sym->release();
1545 }
1546 sym = OSSymbol::withCStringNoCopy(kIOHibernateMemorySignatureEnvKey);
1547 if (sym)
1548 {
1549 gIOOptionsEntry->removeProperty(sym);
1550 sym->release();
1551 }
1552 }
1553 if (data)
1554 data->release();
1555
1556 if (gIOOptionsEntry)
1557 {
1558 if (!vars->haveFastBoot)
1559 {
1560 // reset boot audio volume
1561 IODTPlatformExpert * platform = OSDynamicCast(IODTPlatformExpert, IOService::getPlatform());
1562 if (platform)
1563 platform->writeXPRAM(kXPRamAudioVolume,
1564 &vars->saveBootAudioVolume, sizeof(vars->saveBootAudioVolume));
1565 }
1566
1567 // sync now to hardware if the booter has not
1568 if (kIOHibernateStateInactive == gIOHibernateState)
1569 gIOOptionsEntry->sync();
1570 else
1571 // just sync the variables in case a later panic syncs nvram (it won't sync variables)
1572 gIOOptionsEntry->syncOFVariables();
1573 }
0c530ab8
A		 1574#endif
1575
b0d623f7 1576#if defined(__i386__) || defined(__x86_64__)
2d21ac55
A		 1577 IOService::getPMRootDomain()->removeProperty(kIOHibernateRTCVariablesKey);
1578
1579 /*
1580 * Hibernate variable is written to NVRAM on platforms in which RtcRam
1581 * is not backed by coin cell. Remove Hibernate data from NVRAM.
1582 */
1583 if (gIOOptionsEntry) {
1584 const OSSymbol * sym = OSSymbol::withCStringNoCopy(kIOHibernateRTCVariablesKey);
1585
1586 if (sym) {
4a3eedf9
A		 1587 if (gIOOptionsEntry->getProperty(sym)) {
1588 gIOOptionsEntry->removeProperty(sym);
1589 gIOOptionsEntry->sync();
1590 }
2d21ac55
A		 1591 sym->release();
1592 }
1593 }
0c530ab8 1594#endif
3a60a9f5
A		 1595
1596 if (vars->srcBuffer)
1597 vars->srcBuffer->release();
1598 if (vars->ioBuffer)
1599 vars->ioBuffer->release();
1600 if (vars->fileExtents)
1601 vars->fileExtents->release();
1602
1603 bzero(vars, sizeof(*vars));
1604
1605// gIOHibernateState = kIOHibernateStateInactive; // leave it for post wake code to see
1606
1607 return (kIOReturnSuccess);
1608}
1609
1610IOReturn
1611IOHibernateSystemPostWake(void)
1612{
1613 if (gIOHibernateFileRef)
1614 {
0c530ab8
A		 1615 // invalidate the image file
1616 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderInvalidSignature;
1617 int err = kern_write_file(gIOHibernateFileRef, 0,
1618 (caddr_t) gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
1619 if (KERN_SUCCESS != err)
1620 HIBLOG("kern_write_file(%d)\n", err);
1621
3a60a9f5
A		 1622 kern_close_file_for_direct_io(gIOHibernateFileRef);
1623 gIOHibernateFileRef = 0;
1624 }
1625 return (kIOReturnSuccess);
1626}
1627
1628/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1629
2d21ac55
A		 1630SYSCTL_STRING(_kern, OID_AUTO, hibernatefile,
1631 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1632 gIOHibernateFilename, sizeof(gIOHibernateFilename), "");
1633SYSCTL_STRING(_kern, OID_AUTO, bootsignature,
1634 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1635 gIOHibernateBootSignature, sizeof(gIOHibernateBootSignature), "");
1636SYSCTL_UINT(_kern, OID_AUTO, hibernatemode,
1637 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1638 &gIOHibernateMode, 0, "");
1639
3a60a9f5
A		 1640void
1641IOHibernateSystemInit(IOPMrootDomain * rootDomain)
1642{
1643 OSData * data = OSData::withBytesNoCopy(&gIOHibernateState, sizeof(gIOHibernateState));
1644 if (data)
1645 {
1646 rootDomain->setProperty(kIOHibernateStateKey, data);
1647 data->release();
1648 }
1649
2d21ac55 1650 if (PE_parse_boot_argn("hfile", gIOHibernateFilename, sizeof(gIOHibernateFilename)))
3a60a9f5
A		 1651 gIOHibernateMode = kIOHibernateModeOn;
1652 else
1653 gIOHibernateFilename[0] = 0;
1654
3a60a9f5 1655 sysctl_register_oid(&sysctl__kern_hibernatefile);
3a60a9f5 1656 sysctl_register_oid(&sysctl__kern_bootsignature);
3a60a9f5
A		 1657 sysctl_register_oid(&sysctl__kern_hibernatemode);
1658}
1659
2d21ac55 1660
3a60a9f5
A		 1661/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1662
1663static void
1664hibernate_setup_for_wake(void)
1665{
1666#if __ppc__
1667 // go slow (state needed for wake)
1668 ml_set_processor_speed(1);
1669#endif
1670}
1671
1672/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1673
0c530ab8
A		 1674#define C_ASSERT(e) typedef char __C_ASSERT__[(e) ? 1 : -1]
1675
2d21ac55 1676extern "C" uint32_t
3a60a9f5
A		 1677hibernate_write_image(void)
1678{
1679 IOHibernateImageHeader * header = gIOHibernateCurrentHeader;
1680 IOHibernateVars * vars = &gIOHibernateVars;
1681 IOPolledFileExtent * fileExtents;
1682
0c530ab8
A		 1683 C_ASSERT(sizeof(IOHibernateImageHeader) == 512);
1684
3a60a9f5
A		 1685 uint32_t pageCount, pagesDone;
1686 IOReturn err;
1687 vm_offset_t ppnum;
1688 IOItemCount page, count;
1689 uint8_t * src;
1690 uint8_t * data;
1691 IOByteCount pageCompressedSize;
1692 uint64_t compressedSize, uncompressedSize;
1693 uint64_t image1Size = 0;
1694 uint32_t bitmap_size;
1695 bool iterDone, pollerOpen, needEncryptStart;
1696 uint32_t restore1Sum, sum, sum1, sum2;
1697 uint32_t tag;
1698 uint32_t pageType;
1699 uint32_t pageAndCount[2];
1700
1701 AbsoluteTime startTime, endTime;
1702 AbsoluteTime allTime, compTime, decoTime;
1703 uint64_t nsec;
1704 uint32_t lastProgressStamp = 0;
1705 uint32_t progressStamp;
2d21ac55 1706 uint32_t blob, lastBlob = (uint32_t) -1L;
3a60a9f5
A		 1707
1708 hibernate_cryptvars_t _cryptvars;
1709 hibernate_cryptvars_t * cryptvars = 0;
1710
1711 if (!vars->fileVars || !vars->fileVars->pollers || !vars->fileExtents)
1712 return (false /* sleep */ );
1713
1714 restore1Sum = sum1 = sum2 = 0;
1715
2d21ac55 1716#if CRYPTO
3a60a9f5
A		 1717 // encryption data. "iv" is the "initial vector".
1718 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1719 {
1720 static const unsigned char first_iv[AES_BLOCK_SIZE]
1721 = { 0xa3, 0x63, 0x65, 0xa9, 0x0b, 0x71, 0x7b, 0x1c,
1722 0xdf, 0x9e, 0x5f, 0x32, 0xd7, 0x61, 0x63, 0xda };
1723
1724 cryptvars = &gIOHibernateCryptWakeContext;
1725 bzero(cryptvars, sizeof(hibernate_cryptvars_t));
1726 aes_encrypt_key(vars->cryptKey,
1727 kIOHibernateAESKeySize,
1728 &cryptvars->ctx.encrypt);
1729 aes_decrypt_key(vars->cryptKey,
1730 kIOHibernateAESKeySize,
1731 &cryptvars->ctx.decrypt);
1732
1733 cryptvars = &_cryptvars;
1734 bzero(cryptvars, sizeof(hibernate_cryptvars_t));
1735 aes_encrypt_key(vars->wiredCryptKey,
1736 kIOHibernateAESKeySize,
1737 &cryptvars->ctx.encrypt);
1738
1739 bcopy(&first_iv[0], &cryptvars->aes_iv[0], AES_BLOCK_SIZE);
1740 bzero(&vars->wiredCryptKey[0], sizeof(vars->wiredCryptKey));
1741 bzero(&vars->cryptKey[0], sizeof(vars->cryptKey));
1742 bzero(gIOHibernateCryptWakeVars, sizeof(hibernate_cryptwakevars_t));
1743 }
2d21ac55 1744#endif /* CRYPTO */
3a60a9f5
A		 1745
1746 hibernate_setup_for_wake();
1747
1748 hibernate_page_list_setall(vars->page_list,
1749 vars->page_list_wired,
1750 &pageCount);
1751
1752 HIBLOG("hibernate_page_list_setall found pageCount %d\n", pageCount);
1753
1754 fileExtents = (IOPolledFileExtent *) vars->fileExtents->getBytesNoCopy();
1755
1756#if 0
1757 count = vars->fileExtents->getLength() / sizeof(IOPolledFileExtent);
1758 for (page = 0; page < count; page++)
1759 {
1760 HIBLOG("fileExtents[%d] %qx, %qx (%qx)\n", page,
1761 fileExtents[page].start, fileExtents[page].length,
1762 fileExtents[page].start + fileExtents[page].length);
1763 }
1764#endif
1765
1766 needEncryptStart = (0 != (kIOHibernateModeEncrypt & gIOHibernateMode));
1767
1768 AbsoluteTime_to_scalar(&compTime) = 0;
1769 AbsoluteTime_to_scalar(&decoTime) = 0;
1770
1771 clock_get_uptime(&allTime);
b0d623f7
A		 1772 IOService::getPMRootDomain()->pmStatsRecordEvent(
1773 kIOPMStatsHibernateImageWrite | kIOPMStatsEventStartFlag, allTime);
3a60a9f5
A		 1774
1775 do
1776 {
1777 compressedSize = 0;
1778 uncompressedSize = 0;
1779 iterDone = false;
1780 pageType = 0; // wired pages first
1781
1782 IOPolledFileSeek(vars->fileVars, sizeof(IOHibernateImageHeader));
1783
1784 HIBLOG("IOHibernatePollerOpen, ml_get_interrupts_enabled %d\n",
1785 ml_get_interrupts_enabled());
1786 err = IOHibernatePollerOpen(vars->fileVars, kIOPolledBeforeSleepState, vars->ioBuffer);
1787 HIBLOG("IOHibernatePollerOpen(%x)\n", err);
1788 pollerOpen = (kIOReturnSuccess == err);
1789 if (!pollerOpen)
1790 break;
1791
1792 // copy file block extent list if larger than header
1793
1794 count = vars->fileExtents->getLength();
1795 if (count > sizeof(header->fileExtentMap))
1796 {
1797 count -= sizeof(header->fileExtentMap);
1798 err = IOPolledFileWrite(vars->fileVars,
1799 ((uint8_t *) &fileExtents[0]) + sizeof(header->fileExtentMap), count, cryptvars);
1800 if (kIOReturnSuccess != err)
1801 break;
1802 }
1803
b0d623f7
A		 1804 uintptr_t hibernateBase;
1805 uintptr_t hibernateEnd;
1806
1807#if defined(__i386__) || defined(__x86_64__)
1808 hibernateBase = sectINITPTB;
1809#else
1810 hibernateBase = sectHIBB;
1811#endif
1812
1813 hibernateEnd = (sectHIBB + sectSizeHIB);
3a60a9f5
A		 1814 // copy out restore1 code
1815
b0d623f7
A		 1816 page = atop_32(hibernateBase);
1817 count = atop_32(round_page(hibernateEnd)) - page;
3a60a9f5
A		 1818 header->restore1CodePage = page;
1819 header->restore1PageCount = count;
b0d623f7
A		 1820 header->restore1CodeOffset = ((uintptr_t) &hibernate_machine_entrypoint) - hibernateBase;
1821 header->restore1StackOffset = ((uintptr_t) &gIOHibernateRestoreStackEnd[0]) - 64 - hibernateBase;
3a60a9f5
A		 1822
1823 // sum __HIB sect, with zeros for the stack
b0d623f7 1824 src = (uint8_t *) trunc_page(hibernateBase);
3a60a9f5
A		 1825 for (page = 0; page < count; page++)
1826 {
1827 if ((src < &gIOHibernateRestoreStack[0]) || (src >= &gIOHibernateRestoreStackEnd[0]))
1828 restore1Sum += hibernate_sum(src, page_size);
1829 else
1830 restore1Sum += 0x10000001;
1831 src += page_size;
1832 }
1833 sum1 = restore1Sum;
1834
1835 // write the __HIB sect, with zeros for the stack
1836
b0d623f7
A		 1837 src = (uint8_t *) trunc_page(hibernateBase);
1838 count = ((uintptr_t) &gIOHibernateRestoreStack[0]) - trunc_page(hibernateBase);
3a60a9f5
A		 1839 if (count)
1840 {
1841 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1842 if (kIOReturnSuccess != err)
1843 break;
1844 }
1845 err = IOPolledFileWrite(vars->fileVars,
1846 (uint8_t *) 0,
1847 &gIOHibernateRestoreStackEnd[0] - &gIOHibernateRestoreStack[0],
1848 cryptvars);
1849 if (kIOReturnSuccess != err)
1850 break;
1851 src = &gIOHibernateRestoreStackEnd[0];
b0d623f7 1852 count = round_page(hibernateEnd) - ((uintptr_t) src);
3a60a9f5
A		 1853 if (count)
1854 {
1855 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1856 if (kIOReturnSuccess != err)
1857 break;
1858 }
1859
1860 // write the preview buffer
1861
1862 addr64_t phys64;
1863 IOByteCount segLen;
1864
2d21ac55 1865 if (vars->previewBuffer)
3a60a9f5
A		 1866 {
1867 ppnum = 0;
1868 count = 0;
1869 do
1870 {
b0d623f7 1871 phys64 = vars->previewBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone);
3a60a9f5
A		 1872 pageAndCount[0] = atop_64(phys64);
1873 pageAndCount[1] = atop_32(segLen);
1874 err = IOPolledFileWrite(vars->fileVars,
1875 (const uint8_t *) &pageAndCount, sizeof(pageAndCount),
1876 cryptvars);
1877 if (kIOReturnSuccess != err)
1878 break;
1879 count += segLen;
1880 ppnum += sizeof(pageAndCount);
1881 }
1882 while (phys64);
1883 if (kIOReturnSuccess != err)
1884 break;
1885
b0d623f7 1886 src = (uint8_t *) vars->previewBuffer->getPhysicalSegment(0, NULL, _kIOMemorySourceSegment);
2d21ac55 1887 count = vars->previewBuffer->getLength();
3a60a9f5
A		 1888
1889 header->previewPageListSize = ppnum;
1890 header->previewSize = count + ppnum;
1891
1892 for (page = 0; page < count; page += page_size)
1893 sum1 += hibernate_sum(src + page, page_size);
1894
1895 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1896 if (kIOReturnSuccess != err)
1897 break;
1898 }
1899
1900 // mark areas for no save
1901
1902 for (count = 0;
b0d623f7 1903 (phys64 = vars->ioBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 1904 count += segLen)
1905 {
1906 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1907 atop_64(phys64), atop_32(segLen),
1908 kIOHibernatePageStateFree);
1909 pageCount -= atop_32(segLen);
1910 }
1911
1912 for (count = 0;
b0d623f7 1913 (phys64 = vars->srcBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 1914 count += segLen)
1915 {
1916 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1917 atop_64(phys64), atop_32(segLen),
1918 kIOHibernatePageStateFree);
1919 pageCount -= atop_32(segLen);
1920 }
1921
1922 // copy out bitmap of pages available for trashing during restore
1923
1924 bitmap_size = vars->page_list_wired->list_size;
1925 src = (uint8_t *) vars->page_list_wired;
1926 err = IOPolledFileWrite(vars->fileVars, src, bitmap_size, cryptvars);
1927 if (kIOReturnSuccess != err)
1928 break;
1929
1930 // mark more areas for no save, but these are not available
1931 // for trashing during restore
0c530ab8
A		 1932
1933 hibernate_page_list_set_volatile(vars->page_list, vars->page_list_wired, &pageCount);
3a60a9f5 1934
b0d623f7
A		 1935 page = atop_32(hibernateBase);
1936 count = atop_32(round_page(hibernateEnd)) - page;
3a60a9f5
A		 1937 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1938 page, count,
1939 kIOHibernatePageStateFree);
1940 pageCount -= count;
3a60a9f5
A		 1941
1942 if (vars->previewBuffer) for (count = 0;
b0d623f7 1943 (phys64 = vars->previewBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 1944 count += segLen)
1945 {
1946 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1947 atop_64(phys64), atop_32(segLen),
1948 kIOHibernatePageStateFree);
1949 pageCount -= atop_32(segLen);
1950 }
1951
1952 src = (uint8_t *) vars->srcBuffer->getBytesNoCopy();
1953
0c530ab8
A		 1954 ppnum = 0;
1955 pagesDone = 0;
2d21ac55 1956 lastBlob = 0;
3a60a9f5
A		 1957
1958 HIBLOG("writing %d pages\n", pageCount);
1959
1960 do
1961 {
1962 count = hibernate_page_list_iterate(pageType ? vars->page_list : vars->page_list_wired,
0c530ab8 1963 &ppnum);
3a60a9f5
A		 1964// kprintf("[%d](%x : %x)\n", pageType, ppnum, count);
1965
1966 iterDone = !count;
1967
1968 pageAndCount[0] = ppnum;
1969 pageAndCount[1] = count;
1970 err = IOPolledFileWrite(vars->fileVars,
1971 (const uint8_t *) &pageAndCount, sizeof(pageAndCount),
1972 cryptvars);
1973 if (kIOReturnSuccess != err)
1974 break;
1975
1976 for (page = 0; page < count; page++)
1977 {
1978 err = IOMemoryDescriptorWriteFromPhysical(vars->srcBuffer, 0, ptoa_64(ppnum), page_size);
1979 if (err)
1980 {
b0d623f7 1981 HIBLOG("IOMemoryDescriptorWriteFromPhysical %d [%ld] %x\n", __LINE__, (long)ppnum, err);
3a60a9f5
A		 1982 break;
1983 }
1984
1985 sum = hibernate_sum(src, page_size);
1986
1987 clock_get_uptime(&startTime);
1988
1989 pageCompressedSize = WKdm_compress ((WK_word*) src, (WK_word*) (src + page_size), PAGE_SIZE_IN_WORDS);
1990
1991 clock_get_uptime(&endTime);
1992 ADD_ABSOLUTETIME(&compTime, &endTime);
1993 SUB_ABSOLUTETIME(&compTime, &startTime);
1994
1995 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1996 pageCompressedSize = (pageCompressedSize + AES_BLOCK_SIZE - 1) & ~(AES_BLOCK_SIZE - 1);
1997
1998 if (pageCompressedSize > page_size)
1999 {
2000// HIBLOG("------------lose: %d\n", pageCompressedSize);
2001 pageCompressedSize = page_size;
2002 }
2003
2004 if (pageCompressedSize != page_size)
2005 data = (src + page_size);
2006 else
2007 data = src;
2008
2009 tag = pageCompressedSize | kIOHibernateTagSignature;
2010
2011 if (pageType)
2012 sum2 += sum;
2013 else
2014 sum1 += sum;
2015
2016 if (needEncryptStart && (ppnum >= atop_32(sectDATAB)))
2017 {
2018 // start encrypting partway into the data about to be written
2019 vars->fileVars->encryptStart = (vars->fileVars->position + AES_BLOCK_SIZE - 1)
2020 & ~(AES_BLOCK_SIZE - 1);
2021 needEncryptStart = false;
2022 }
2023
2024 err = IOPolledFileWrite(vars->fileVars, (const uint8_t *) &tag, sizeof(tag), cryptvars);
2025 if (kIOReturnSuccess != err)
2026 break;
2027
2028 err = IOPolledFileWrite(vars->fileVars, data, (pageCompressedSize + 3) & ~3, cryptvars);
2029 if (kIOReturnSuccess != err)
2030 break;
2031
2032 compressedSize += pageCompressedSize;
2033 if (pageCompressedSize)
2034 uncompressedSize += page_size;
2035 ppnum++;
2036 pagesDone++;
2d21ac55
A		 2037
2038 if (vars->consoleMapping && (0 == (1023 & pagesDone)))
2039 {
2040 blob = ((pagesDone * kIOHibernateProgressCount) / pageCount);
2041 if (blob != lastBlob)
2042 {
2043 ProgressUpdate(gIOHibernateGraphicsInfo, vars->consoleMapping, lastBlob, blob);
2044 lastBlob = blob;
2045 }
2046 }
3a60a9f5
A		 2047 if (0 == (8191 & pagesDone))
2048 {
2049 clock_get_uptime(&endTime);
2050 SUB_ABSOLUTETIME(&endTime, &allTime);
2051 absolutetime_to_nanoseconds(endTime, &nsec);
2052 progressStamp = nsec / 750000000ULL;
2053 if (progressStamp != lastProgressStamp)
2054 {
2055 lastProgressStamp = progressStamp;
2056 HIBPRINT("pages %d (%d%%)\n", pagesDone, (100 * pagesDone) / pageCount);
2057 }
2058 }
2059 }
2060 if (kIOReturnSuccess != err)
2061 break;
2062 if (iterDone && !pageType)
2063 {
2064 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
2065 if (kIOReturnSuccess != err)
2066 break;
2067
2068 iterDone = false;
2069 pageType = 1;
0c530ab8 2070 ppnum = 0;
3a60a9f5
A		 2071 image1Size = vars->fileVars->position;
2072 if (cryptvars)
2073 {
2074 bcopy(&cryptvars->aes_iv[0],
2075 &gIOHibernateCryptWakeContext.aes_iv[0],
2076 sizeof(cryptvars->aes_iv));
2077 cryptvars = &gIOHibernateCryptWakeContext;
2078 }
2079 HIBLOG("image1Size %qd\n", image1Size);
2080 }
2081 }
2082 while (!iterDone);
2083 if (kIOReturnSuccess != err)
2084 break;
2085 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
2086 if (kIOReturnSuccess != err)
2087 break;
2088
2089 // Header:
2090
2091 header->imageSize = vars->fileVars->position;
2092 header->image1Size = image1Size;
2093 header->bitmapSize = bitmap_size;
2094 header->pageCount = pageCount;
2095 header->encryptStart = vars->fileVars->encryptStart;
2096
2097 header->restore1Sum = restore1Sum;
2098 header->image1Sum = sum1;
2099 header->image2Sum = sum2;
2100
2101 count = vars->fileExtents->getLength();
2102 if (count > sizeof(header->fileExtentMap))
2103 {
2104 header->fileExtentMapSize = count;
2105 count = sizeof(header->fileExtentMap);
2106 }
2107 else
2108 header->fileExtentMapSize = sizeof(header->fileExtentMap);
2109 bcopy(&fileExtents[0], &header->fileExtentMap[0], count);
2110
2111 IOPolledFileSeek(vars->fileVars, 0);
2112 err = IOPolledFileWrite(vars->fileVars,
2113 (uint8_t *) header, sizeof(IOHibernateImageHeader),
2114 cryptvars);
2115 if (kIOReturnSuccess != err)
2116 break;
2117 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
2118 if (kIOReturnSuccess != err)
2119 break;
2d21ac55 2120 err = IOHibernatePollerIODone(vars->fileVars, true);
3a60a9f5
A		 2121 if (kIOReturnSuccess != err)
2122 break;
2123 }
2124 while (false);
2125
2126 clock_get_uptime(&endTime);
b0d623f7
A		 2127
2128 IOService::getPMRootDomain()->pmStatsRecordEvent(
2129 kIOPMStatsHibernateImageWrite | kIOPMStatsEventStopFlag, endTime);
2130
3a60a9f5
A		 2131 SUB_ABSOLUTETIME(&endTime, &allTime);
2132 absolutetime_to_nanoseconds(endTime, &nsec);
2133 HIBLOG("all time: %qd ms, ",
2134 nsec / 1000000ULL);
2135
2136 absolutetime_to_nanoseconds(compTime, &nsec);
2137 HIBLOG("comp time: %qd ms, ",
2138 nsec / 1000000ULL);
2139
2140 absolutetime_to_nanoseconds(decoTime, &nsec);
2141 HIBLOG("deco time: %qd ms, ",
2142 nsec / 1000000ULL);
2143
2144 HIBLOG("\nimage %qd, uncompressed %qd (%d), compressed %qd (%d%%), sum1 %x, sum2 %x\n",
2145 header->imageSize,
2146 uncompressedSize, atop_32(uncompressedSize), compressedSize,
8f6c56a5 2147 uncompressedSize ? ((int) ((compressedSize * 100ULL) / uncompressedSize)) : 0,
3a60a9f5
A		 2148 sum1, sum2);
2149
c910b4d9
A		 2150 if (vars->fileVars->io)
2151 (void) IOHibernatePollerIODone(vars->fileVars, false);
2152
3a60a9f5
A		 2153 if (pollerOpen)
2154 IOHibernatePollerClose(vars->fileVars, kIOPolledBeforeSleepState);
2155
2d21ac55
A		 2156 if (vars->consoleMapping)
2157 ProgressUpdate(gIOHibernateGraphicsInfo,
2158 vars->consoleMapping, 0, kIOHibernateProgressCount);
2159
3a60a9f5
A		 2160 HIBLOG("hibernate_write_image done(%x)\n", err);
2161
2162 // should we come back via regular wake, set the state in memory.
2163 gIOHibernateState = kIOHibernateStateInactive;
2164
2d21ac55
A		 2165 if (kIOReturnSuccess == err)
2166 {
2167 if (kIOHibernateModeSleep & gIOHibernateMode)
2168 {
2169 return (kIOHibernatePostWriteSleep);
2170 }
2171 else if(kIOHibernateModeRestart & gIOHibernateMode)
2172 {
2173 return (kIOHibernatePostWriteRestart);
2174 }
2175 else
2176 {
2177 /* by default, power down */
2178 return (kIOHibernatePostWriteHalt);
2179 }
2180 }
2181 else if (kIOReturnAborted == err)
2182 {
2183 return (kIOHibernatePostWriteWake);
2184 }
3a60a9f5 2185 else
3a60a9f5 2186 {
2d21ac55
A		 2187 /* on error, sleep */
2188 return (kIOHibernatePostWriteSleep);
3a60a9f5
A		 2189 }
2190}
2191
2192/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
2193
2194extern "C" void
2195hibernate_machine_init(void)
2196{
2197 IOReturn err;
2198 uint32_t sum;
2199 uint32_t pagesDone;
2200 AbsoluteTime allTime, endTime;
2201 uint64_t nsec;
2202 uint32_t lastProgressStamp = 0;
2203 uint32_t progressStamp;
2204 uint64_t progressZeroPosition = 0;
2205 uint32_t blob, lastBlob = (uint32_t) -1L;
2206 hibernate_cryptvars_t * cryptvars = 0;
2207
2208 IOHibernateVars * vars = &gIOHibernateVars;
2209
2210 if (!vars->fileVars || !vars->fileVars->pollers || !vars->fileExtents)
2211 return;
2212
3a60a9f5
A		 2213 sum = gIOHibernateCurrentHeader->actualImage1Sum;
2214 pagesDone = gIOHibernateCurrentHeader->actualUncompressedPages;
2215
2216 HIBLOG("hibernate_machine_init: state %d, image pages %d, sum was %x, image1Size %qx, conflictCount %d, nextFree %x\n",
2217 gIOHibernateState, pagesDone, sum, gIOHibernateCurrentHeader->image1Size,
2218 gIOHibernateCurrentHeader->conflictCount, gIOHibernateCurrentHeader->nextFree);
2219
2220 if (kIOHibernateStateWakingFromHibernate != gIOHibernateState)
2221 {
2222 HIBLOG("regular wake\n");
2223 return;
2224 }
2225
2226 HIBPRINT("diag %x %x %x %x\n",
2227 gIOHibernateCurrentHeader->diag[0], gIOHibernateCurrentHeader->diag[1],
2228 gIOHibernateCurrentHeader->diag[2], gIOHibernateCurrentHeader->diag[3]);
2229
0c530ab8 2230 HIBPRINT("video %x %d %d %d\n",
3a60a9f5
A		 2231 gIOHibernateGraphicsInfo->physicalAddress, gIOHibernateGraphicsInfo->depth,
2232 gIOHibernateGraphicsInfo->width, gIOHibernateGraphicsInfo->height);
2233
0c530ab8
A		 2234 if ((kIOHibernateModeDiscardCleanActive | kIOHibernateModeDiscardCleanInactive) & gIOHibernateMode)
2235 hibernate_page_list_discard(vars->page_list);
2236
2237 boot_args *args = (boot_args *) PE_state.bootArgs;
2238
2239 if (vars->videoMapping
2240 && gIOHibernateGraphicsInfo->physicalAddress
2241 && (args->Video.v_baseAddr == gIOHibernateGraphicsInfo->physicalAddress))
3a60a9f5
A		 2242 {
2243 vars->videoMapSize = round_page(gIOHibernateGraphicsInfo->height
2244 * gIOHibernateGraphicsInfo->rowBytes);
2245 IOMapPages(kernel_map,
2246 vars->videoMapping, gIOHibernateGraphicsInfo->physicalAddress,
2247 vars->videoMapSize, kIOMapInhibitCache );
2248 }
2249
0c530ab8
A		 2250 uint8_t * src = (uint8_t *) vars->srcBuffer->getBytesNoCopy();
2251
2252 if (gIOHibernateWakeMapSize)
2253 {
2254 err = IOMemoryDescriptorWriteFromPhysical(vars->srcBuffer, 0, ptoa_64(gIOHibernateWakeMap),
2255 gIOHibernateWakeMapSize);
2256 if (kIOReturnSuccess == err)
2257 hibernate_newruntime_map(src, gIOHibernateWakeMapSize,
2258 gIOHibernateCurrentHeader->systemTableOffset);
2259 gIOHibernateWakeMap = 0;
2260 gIOHibernateWakeMapSize = 0;
2261 }
2262
3a60a9f5
A		 2263 uint32_t decoOffset;
2264
2265 clock_get_uptime(&allTime);
2266
2267 HIBLOG("IOHibernatePollerOpen(), ml_get_interrupts_enabled %d\n", ml_get_interrupts_enabled());
2268 err = IOHibernatePollerOpen(vars->fileVars, kIOPolledAfterSleepState, 0);
2269 HIBLOG("IOHibernatePollerOpen(%x)\n", err);
2270
2271 if (gIOHibernateCurrentHeader->previewSize)
2272 progressZeroPosition = gIOHibernateCurrentHeader->previewSize
2273 + gIOHibernateCurrentHeader->fileExtentMapSize
2274 - sizeof(gIOHibernateCurrentHeader->fileExtentMap)
2275 + ptoa_64(gIOHibernateCurrentHeader->restore1PageCount);
2276
2277 IOPolledFileSeek(vars->fileVars, gIOHibernateCurrentHeader->image1Size);
2278
2d21ac55 2279 if (vars->videoMapSize)
3a60a9f5
A		 2280 {
2281 lastBlob = ((vars->fileVars->position - progressZeroPosition) * kIOHibernateProgressCount)
2282 / (gIOHibernateCurrentHeader->imageSize - progressZeroPosition);
2283 ProgressUpdate(gIOHibernateGraphicsInfo, (uint8_t *) vars->videoMapping, 0, lastBlob);
2284 }
2285
2286 cryptvars = (kIOHibernateModeEncrypt & gIOHibernateMode) ? &gIOHibernateCryptWakeContext : 0;
2287 if (kIOHibernateModeEncrypt & gIOHibernateMode)
2288 {
2289 cryptvars = &gIOHibernateCryptWakeContext;
2290 bcopy(&gIOHibernateCryptWakeVars->aes_iv[0],
2291 &cryptvars->aes_iv[0],
2292 sizeof(cryptvars->aes_iv));
2293 }
2294
2295 // kick off the read ahead
2296 vars->fileVars->io = false;
2297 vars->fileVars->bufferHalf = 0;
2298 vars->fileVars->bufferLimit = 0;
2299 vars->fileVars->lastRead = 0;
2300 vars->fileVars->bufferOffset = vars->fileVars->bufferLimit;
2301
2302 IOPolledFileRead(vars->fileVars, 0, 0, cryptvars);
2303 vars->fileVars->bufferOffset = vars->fileVars->bufferLimit;
2304 // --
2305
2306 HIBLOG("hibernate_machine_init reading\n");
2307
2308 uint32_t * header = (uint32_t *) src;
2309 sum = 0;
2310
2311 do
2312 {
2313 unsigned int count;
2314 unsigned int page;
2315 uint32_t tag;
2316 vm_offset_t ppnum, compressedSize;
2317
36401178
A		 2318 err = IOPolledFileRead(vars->fileVars, src, 8, cryptvars);
2319 if (kIOReturnSuccess != err)
2320 break;
3a60a9f5
A		 2321
2322 ppnum = header[0];
2323 count = header[1];
2324
2325// HIBPRINT("(%x, %x)\n", ppnum, count);
2326
2327 if (!count)
2328 break;
2329
2330 for (page = 0; page < count; page++)
2331 {
36401178
A		 2332 err = IOPolledFileRead(vars->fileVars, (uint8_t *) &tag, 4, cryptvars);
2333 if (kIOReturnSuccess != err)
2334 break;
3a60a9f5
A		 2335
2336 compressedSize = kIOHibernateTagLength & tag;
36401178
A		 2337 if (kIOHibernateTagSignature != (tag & ~kIOHibernateTagLength))
2338 {
2339 err = kIOReturnIPCError;
2340 break;
2341 }
2342
3a60a9f5
A		 2343 if (!compressedSize)
2344 {
2345 ppnum++;
2346 pagesDone++;
2347 continue;
2348 }
2349
36401178
A		 2350 err = IOPolledFileRead(vars->fileVars, src, (compressedSize + 3) & ~3, cryptvars);
2351 if (kIOReturnSuccess != err)
2352 break;
2353
2354 if (compressedSize < page_size)
3a60a9f5
A		 2355 {
2356 decoOffset = page_size;
2357 WKdm_decompress((WK_word*) src, (WK_word*) (src + decoOffset), PAGE_SIZE_IN_WORDS);
2358 }
2359 else
2360 decoOffset = 0;
2361
2362 sum += hibernate_sum((src + decoOffset), page_size);
2363
2364 err = IOMemoryDescriptorReadToPhysical(vars->srcBuffer, decoOffset, ptoa_64(ppnum), page_size);
2365 if (err)
36401178 2366 {
b0d623f7 2367 HIBLOG("IOMemoryDescriptorReadToPhysical [%ld] %x\n", (long)ppnum, err);
36401178
A		 2368 break;
2369 }
3a60a9f5
A		 2370
2371 ppnum++;
2372 pagesDone++;
2373
2d21ac55 2374 if (vars->videoMapSize && (0 == (1023 & pagesDone)))
3a60a9f5
A		 2375 {
2376 blob = ((vars->fileVars->position - progressZeroPosition) * kIOHibernateProgressCount)
2377 / (gIOHibernateCurrentHeader->imageSize - progressZeroPosition);
2378 if (blob != lastBlob)
2379 {
2380 ProgressUpdate(gIOHibernateGraphicsInfo, (uint8_t *) vars->videoMapping, lastBlob, blob);
2381 lastBlob = blob;
2382 }
2383 }
2384
2385 if (0 == (8191 & pagesDone))
2386 {
2387 clock_get_uptime(&endTime);
2388 SUB_ABSOLUTETIME(&endTime, &allTime);
2389 absolutetime_to_nanoseconds(endTime, &nsec);
2390 progressStamp = nsec / 750000000ULL;
2391 if (progressStamp != lastProgressStamp)
2392 {
2393 lastProgressStamp = progressStamp;
2394 HIBPRINT("pages %d (%d%%)\n", pagesDone,
2395 (100 * pagesDone) / gIOHibernateCurrentHeader->pageCount);
2396 }
2397 }
2398 }
2399 }
2400 while (true);
2401
36401178
A		 2402 if (kIOReturnSuccess != err)
2403 panic("Hibernate restore error %x", err);
2404
3a60a9f5
A		 2405 gIOHibernateCurrentHeader->actualImage2Sum = sum;
2406
2407 if (vars->fileVars->io)
2d21ac55 2408 (void) IOHibernatePollerIODone(vars->fileVars, false);
3a60a9f5
A		 2409
2410 err = IOHibernatePollerClose(vars->fileVars, kIOPolledAfterSleepState);
2411
2d21ac55 2412 if (vars->videoMapSize)
3a60a9f5
A		 2413 ProgressUpdate(gIOHibernateGraphicsInfo,
2414 (uint8_t *) vars->videoMapping, 0, kIOHibernateProgressCount);
2415
2416 clock_get_uptime(&endTime);
b0d623f7
A		 2417
2418 IOService::getPMRootDomain()->pmStatsRecordEvent(
2419 kIOPMStatsHibernateImageRead | kIOPMStatsEventStartFlag, allTime);
2420 IOService::getPMRootDomain()->pmStatsRecordEvent(
2421 kIOPMStatsHibernateImageRead | kIOPMStatsEventStopFlag, endTime);
2422
3a60a9f5
A		 2423 SUB_ABSOLUTETIME(&endTime, &allTime);
2424 absolutetime_to_nanoseconds(endTime, &nsec);
2425
2426 HIBLOG("hibernate_machine_init pagesDone %d sum2 %x, time: %qd ms\n",
2427 pagesDone, sum, nsec / 1000000ULL);
2428}
2429
2430/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
mirror of XNU