]> git.saurik.com Git - apple/xnu.git/blame - iokit/Kernel/IOHibernateIO.cpp
projects / apple / xnu.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
xnu-792.6.70.tar.gz
[apple/xnu.git] / iokit / Kernel / IOHibernateIO.cpp
CommitLineData
3a60a9f5
A		 1/*
2 * Copyright (c) 2004 Apple Computer, Inc. All rights reserved.
3 *
4 * @APPLE_LICENSE_HEADER_START@
5 *
37839358
A		 6 * The contents of this file constitute Original Code as defined in and
7 * are subject to the Apple Public Source License Version 1.1 (the
8 * "License"). You may not use this file except in compliance with the
9 * License. Please obtain a copy of the License at
10 * http://www.apple.com/publicsource and read it before using this file.
3a60a9f5 11 *
37839358
A		 12 * This Original Code and all software distributed under the License are
13 * distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, EITHER
3a60a9f5
A		 14 * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
15 * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
37839358
A		 16 * FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. Please see the
17 * License for the specific language governing rights and limitations
18 * under the License.
3a60a9f5
A		 19 *
20 * @APPLE_LICENSE_HEADER_END@
21 */
22
23
24/*
25
26Sleep:
27
28- PMRootDomain calls IOHibernateSystemSleep() before system sleep
29(devices awake, normal execution context)
30- IOHibernateSystemSleep opens the hibernation file (or partition) at the bsd level,
31 grabs its extents and searches for a polling driver willing to work with that IOMedia.
32 The BSD code makes an ioctl to the storage driver to get the partition base offset to
33 the disk, and other ioctls to get the transfer constraints
34 If successful, the file is written to make sure its initially not bootable (in case of
35 later failure) and nvram set to point to the first block of the file. (Has to be done
36 here so blocking is possible in nvram support).
37 hibernate_setup() in osfmk is called to allocate page bitmaps for all dram, and
38 page out any pages it wants to (currently zero, but probably some percentage of memory).
39 Its assumed just allocating pages will cause the VM system to naturally select the best
40 pages for eviction. It also copies processor flags needed for the restore path and sets
41 a flag in the boot processor proc info.
42 gIOHibernateState = kIOHibernateStateHibernating.
43- Regular sleep progresses - some drivers may inspect the root domain property
44 kIOHibernateStateKey to modify behavior. The platform driver saves state to memory
45 as usual but leaves motherboard I/O on.
46- Eventually the platform calls ml_ppc_sleep() in the shutdown context on the last cpu,
47 at which point memory is ready to be saved. mapping_hibernate_flush() is called to get
48 all ppc RC bits out of the hash table and caches into the mapping structures.
49- hibernate_write_image() is called (still in shutdown context, no blocking or preemption).
50 hibernate_page_list_setall() is called to get a bitmap of dram pages that need to be saved.
51 All pages are assumed to be saved (as part of the wired image) unless explicitly subtracted
52 by hibernate_page_list_setall(), avoiding having to find arch dependent low level bits.
53 The image header and block list are written. The header includes the second file extent so
54 only the header block is needed to read the file, regardless of filesystem.
55 The kernel section "__HIB" is written uncompressed to the image. This section of code and data
56 (only) is used to decompress the image during wake/boot.
57 Some additional pages are removed from the bitmaps - the buffers used for hibernation.
58 The bitmaps are written to the image.
59 More areas are removed from the bitmaps (after they have been written to the image) - the
60 section "__HIB" pages and interrupt stack.
61 Each wired page is compressed and written and then each non-wired page. Compression and
62 disk writes are in parallel.
63 The image header is written to the start of the file and the polling driver closed.
64 The machine powers down (or sleeps).
65
66Boot/Wake:
67
68- BootX sees the boot-image nvram variable containing the device and block number of the image,
69 reads the header and if the signature is correct proceeds. The boot-image variable is cleared.
70- BootX reads the portion of the image used for wired pages, to memory. Its assumed this will fit
71 in the OF memory environment, and the image is decrypted. There is no decompression in BootX,
72 that is in the kernel's __HIB section.
73- BootX copies the "__HIB" section to its correct position in memory, quiesces and calls its entry
74 hibernate_kernel_entrypoint(), passing the location of the image in memory. Translation is off,
75 only code & data in that section is safe to call since all the other wired pages are still
76 compressed in the image.
77- hibernate_kernel_entrypoint() removes pages occupied by the raw image from the page bitmaps.
78 It uses the bitmaps to work out which pages can be uncompressed from the image to their final
79 location directly, and copies those that can't to interim free pages. When the image has been
80 completed, the copies are uncompressed, overwriting the wired image pages.
81 hibernate_restore_phys_page() (in osfmk since its arch dependent, but part of the "__HIB" section)
82 is used to get pages into place for 64bit.
83- the reset vector is called (at least on ppc), the kernel proceeds on a normal wake, with some
84 changes conditional on the per proc flag - before VM is turned on the boot cpu, all mappings
85 are removed from the software strutures, and the hash table is reinitialized.
86- After the platform CPU init code is called, hibernate_machine_init() is called to restore the rest
87 of memory, using the polled mode driver, before other threads can run or any devices are turned on.
88 This reduces the memory usage for BootX and allows decompression in parallel with disk reads,
89 for the remaining non wired pages.
90- The polling driver is closed down and regular wake proceeds. When the kernel calls iokit to wake
91 (normal execution context) hibernate_teardown() in osmfk is called to release any memory, the file
92 is closed via bsd.
93
94Polled Mode I/O:
95
96IOHibernateSystemSleep() finds a polled mode interface to the ATA controller via a property in the
97registry, specifying an object of calls IOPolledInterface.
98
99Before the system goes to sleep it searches from the IOMedia object (could be a filesystem or
100partition) that the image is going to live, looking for polled interface properties. If it finds
101one the IOMedia object is passed to a "probe" call for the interface to accept or reject. All the
102interfaces found are kept in an ordered list.
103
104There is an Open/Close pair of calls made to each of the interfaces at various stages since there are
105few different contexts things happen in:
106
107- there is an Open/Close (Preflight) made before any part of the system has slept (I/O is all
108up and running) and after wake - this is safe to allocate memory and do anything. The device
109ignores sleep requests from that point since its a waste of time if it goes to sleep and
110immediately wakes back up for the image write.
111
112- there is an Open/Close (BeforeSleep) pair made around the image write operations that happen
113immediately before sleep. These can't block or allocate memory - the I/O system is asleep apart
114from the low level bits (motherboard I/O etc). There is only one thread running. The close can be
115used to flush and set the disk to sleep.
116
117- there is an Open/Close (AfterSleep) pair made around the image read operations that happen
118immediately after sleep. These can't block or allocate memory. This is happening after the platform
119expert has woken the low level bits of the system, but most of the I/O system has not. There is only
120one thread running.
121
122For the actual I/O, all the ops are with respect to a single IOMemoryDescriptor that was passed
123(prepared) to the Preflight Open() call. There is a read/write op, buffer offset to the IOMD for
124the data, an offset to the disk and length (block aligned 64 bit numbers), and completion callback.
125Each I/O is async but only one is ever outstanding. The polled interface has a checkForWork call
126that is called for the hardware to check for events, and complete the I/O via the callback.
127The hibernate path uses the same transfer constraints the regular cluster I/O path in BSD uses
128to restrict I/O ops.
129*/
130
131#include <sys/systm.h>
132
133#include <IOKit/IOWorkLoop.h>
134#include <IOKit/IOCommandGate.h>
135#include <IOKit/IOTimerEventSource.h>
136#include <IOKit/IOPlatformExpert.h>
137#include <IOKit/IOKitDebug.h>
138#include <IOKit/IOTimeStamp.h>
139#include <IOKit/pwr_mgt/RootDomain.h>
140#include <IOKit/pwr_mgt/IOPMPrivate.h>
141#include <IOKit/IOMessage.h>
142#include <IOKit/IODeviceTreeSupport.h>
143#include <IOKit/IOBSD.h>
144#include "RootDomainUserClient.h"
145#include <IOKit/pwr_mgt/IOPowerConnection.h>
146#include "IOPMPowerStateQueue.h"
147#include <IOKit/IOBufferMemoryDescriptor.h>
148#include <crypto/aes.h>
149
150#include <sys/uio.h>
151#include <sys/conf.h>
152#include <sys/stat.h>
153#include <sys/fcntl.h> // (FWRITE, ...)
154extern "C" {
155#include <sys/sysctl.h>
156}
157
158#include <IOKit/IOHibernatePrivate.h>
159#include <IOKit/IOPolledInterface.h>
160#include <IOKit/IONVRAM.h>
161#include "IOHibernateInternal.h"
162#include "WKdm.h"
163#include "IOKitKernelInternal.h"
164
165/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
166
167OSDefineMetaClassAndAbstractStructors(IOPolledInterface, OSObject);
168
169OSMetaClassDefineReservedUnused(IOPolledInterface, 0);
170OSMetaClassDefineReservedUnused(IOPolledInterface, 1);
171OSMetaClassDefineReservedUnused(IOPolledInterface, 2);
172OSMetaClassDefineReservedUnused(IOPolledInterface, 3);
173OSMetaClassDefineReservedUnused(IOPolledInterface, 4);
174OSMetaClassDefineReservedUnused(IOPolledInterface, 5);
175OSMetaClassDefineReservedUnused(IOPolledInterface, 6);
176OSMetaClassDefineReservedUnused(IOPolledInterface, 7);
177OSMetaClassDefineReservedUnused(IOPolledInterface, 8);
178OSMetaClassDefineReservedUnused(IOPolledInterface, 9);
179OSMetaClassDefineReservedUnused(IOPolledInterface, 10);
180OSMetaClassDefineReservedUnused(IOPolledInterface, 11);
181OSMetaClassDefineReservedUnused(IOPolledInterface, 12);
182OSMetaClassDefineReservedUnused(IOPolledInterface, 13);
183OSMetaClassDefineReservedUnused(IOPolledInterface, 14);
184OSMetaClassDefineReservedUnused(IOPolledInterface, 15);
185
186/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
187
188extern uint32_t gIOHibernateState;
189uint32_t gIOHibernateMode;
190static char gIOHibernateBootSignature[256+1];
191static char gIOHibernateFilename[MAXPATHLEN+1];
192static uint32_t gIOHibernateFreeRatio = 0; // free page target (percent)
193uint32_t gIOHibernateFreeTime = 0*1000; // max time to spend freeing pages (ms)
194
195static IODTNVRAM * gIOOptionsEntry;
196static IORegistryEntry * gIOChosenEntry;
197
198static IOPolledFileIOVars gFileVars;
199static IOHibernateVars gIOHibernateVars;
200static struct kern_direct_file_io_ref_t * gIOHibernateFileRef;
201static hibernate_cryptvars_t gIOHibernateCryptWakeContext;
202
203/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
204
205enum { kXPRamAudioVolume = 8 };
206enum { kDefaultIOSize = 128 * 1024 };
207enum { kVideoMapSize = 32 * 1024 * 1024 };
208
209#ifndef kIOMediaPreferredBlockSizeKey
210#define kIOMediaPreferredBlockSizeKey "Preferred Block Size"
211#endif
212
213#ifndef kIOBootPathKey
214#define kIOBootPathKey "bootpath"
215#endif
216#ifndef kIOSelectedBootDeviceKey
217#define kIOSelectedBootDeviceKey "boot-device"
218#endif
219
220
221enum { kIOHibernateMinPollersNeeded = 2 };
222
223/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
224
225// copy from phys addr to MD
226
227static IOReturn
228IOMemoryDescriptorWriteFromPhysical(IOMemoryDescriptor * md,
229 IOByteCount offset, addr64_t bytes, IOByteCount length)
230{
231 addr64_t srcAddr = bytes;
232 IOByteCount remaining;
233
234 remaining = length = min(length, md->getLength() - offset);
235 while (remaining) { // (process another target segment?)
236 addr64_t dstAddr64;
237 IOByteCount dstLen;
238
239 dstAddr64 = md->getPhysicalSegment64(offset, &dstLen);
240 if (!dstAddr64)
241 break;
242
243 // Clip segment length to remaining
244 if (dstLen > remaining)
245 dstLen = remaining;
246
247#if 1
248 bcopy_phys(srcAddr, dstAddr64, dstLen);
249#else
250 copypv(srcAddr, dstAddr64, dstLen,
251 cppvPsnk | cppvFsnk | cppvNoRefSrc | cppvNoModSnk | cppvKmap);
252#endif
253 srcAddr += dstLen;
254 offset += dstLen;
255 remaining -= dstLen;
256 }
257
258 assert(!remaining);
259
260 return remaining ? kIOReturnUnderrun : kIOReturnSuccess;
261}
262
263// copy from MD to phys addr
264
265static IOReturn
266IOMemoryDescriptorReadToPhysical(IOMemoryDescriptor * md,
267 IOByteCount offset, addr64_t bytes, IOByteCount length)
268{
269 addr64_t dstAddr = bytes;
270 IOByteCount remaining;
271
272 remaining = length = min(length, md->getLength() - offset);
273 while (remaining) { // (process another target segment?)
274 addr64_t srcAddr64;
275 IOByteCount dstLen;
276
277 srcAddr64 = md->getPhysicalSegment64(offset, &dstLen);
278 if (!srcAddr64)
279 break;
280
281 // Clip segment length to remaining
282 if (dstLen > remaining)
283 dstLen = remaining;
284
285#if 1
286 bcopy_phys(srcAddr64, dstAddr, dstLen);
287#else
288 copypv(srcAddr, dstAddr64, dstLen,
289 cppvPsnk | cppvFsnk | cppvNoRefSrc | cppvNoModSnk | cppvKmap);
290#endif
291 dstAddr += dstLen;
292 offset += dstLen;
293 remaining -= dstLen;
294 }
295
296 assert(!remaining);
297
298 return remaining ? kIOReturnUnderrun : kIOReturnSuccess;
299}
300
301/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
302
303void
304hibernate_set_page_state(hibernate_page_list_t * page_list, hibernate_page_list_t * page_list_wired,
305 vm_offset_t ppnum, vm_offset_t count, uint32_t kind)
306{
307 count += ppnum;
308 switch (kind)
309 {
310 case kIOHibernatePageStateUnwiredSave:
311 // unwired save
312 for (; ppnum < count; ppnum++)
313 {
314 hibernate_page_bitset(page_list, FALSE, ppnum);
315 hibernate_page_bitset(page_list_wired, TRUE, ppnum);
316 }
317 break;
318 case kIOHibernatePageStateWiredSave:
319 // wired save
320 for (; ppnum < count; ppnum++)
321 {
322 hibernate_page_bitset(page_list, FALSE, ppnum);
323 hibernate_page_bitset(page_list_wired, FALSE, ppnum);
324 }
325 break;
326 case kIOHibernatePageStateFree:
327 // free page
328 for (; ppnum < count; ppnum++)
329 {
330 hibernate_page_bitset(page_list, TRUE, ppnum);
331 hibernate_page_bitset(page_list_wired, TRUE, ppnum);
332 }
333 break;
334 default:
335 panic("hibernate_set_page_state");
336 }
337}
338
339static vm_offset_t
340hibernate_page_list_iterate(hibernate_page_list_t * list,
341 void ** iterator, vm_offset_t * ppnum)
342{
343 uint32_t count, idx;
344
345 idx = (uint32_t) *iterator;
346
347 if (!idx)
348 idx = hibernate_page_list_count(list, TRUE, idx);
349
350 *ppnum = idx;
351 count = hibernate_page_list_count(list, FALSE, idx);
352 idx += count;
353 idx += hibernate_page_list_count(list, TRUE, idx);
354 *iterator = (void *) idx;
355
356 return (count);
357}
358
359/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
360
361static IOReturn
362IOHibernatePollerProbe(IOPolledFileIOVars * vars, IOService * target)
363{
364 IOReturn err = kIOReturnError;
365 int32_t idx;
366 IOPolledInterface * poller;
367
368 for (idx = vars->pollers->getCount() - 1; idx >= 0; idx--)
369 {
370 poller = (IOPolledInterface *) vars->pollers->getObject(idx);
371 err = poller->probe(target);
372 if (err)
373 {
374 HIBLOG("IOPolledInterface::probe[%d] 0x%x\n", idx, err);
375 break;
376 }
377 }
378
379 return (err);
380}
381
382static IOReturn
383IOHibernatePollerOpen(IOPolledFileIOVars * vars, uint32_t state, IOMemoryDescriptor * md)
384{
385 IOReturn err = kIOReturnError;
386 int32_t idx;
387 IOPolledInterface * poller;
388
389 for (idx = vars->pollers->getCount() - 1; idx >= 0; idx--)
390 {
391 poller = (IOPolledInterface *) vars->pollers->getObject(idx);
392 err = poller->open(state, md);
393 if (err)
394 {
395 HIBLOG("IOPolledInterface::open[%d] 0x%x\n", idx, err);
396 break;
397 }
398 }
399
400 return (err);
401}
402
403static IOReturn
404IOHibernatePollerClose(IOPolledFileIOVars * vars, uint32_t state)
405{
406 IOReturn err = kIOReturnError;
407 int32_t idx;
408 IOPolledInterface * poller;
409
410 for (idx = 0;
411 (poller = (IOPolledInterface *) vars->pollers->getObject(idx));
412 idx++)
413 {
414 err = poller->close(state);
415 if (err)
416 HIBLOG("IOPolledInterface::close[%d] 0x%x\n", idx, err);
417 }
418
419 return (err);
420}
421
422static void
423IOHibernatePollerIOComplete(void * target,
424 void * parameter,
425 IOReturn status,
426 UInt64 actualByteCount)
427{
428 IOPolledFileIOVars * vars = (IOPolledFileIOVars *) parameter;
429
430 vars->ioStatus = status;
431}
432
433static IOReturn
434IOHibernatePollerIO(IOPolledFileIOVars * vars,
435 uint32_t operation, uint32_t bufferOffset,
436 uint64_t deviceOffset, uint64_t length)
437{
438
439 IOReturn err = kIOReturnError;
440 IOPolledInterface * poller;
441 IOPolledCompletion completion;
442
443 completion.target = 0;
444 completion.action = &IOHibernatePollerIOComplete;
445 completion.parameter = vars;
446
447 vars->ioStatus = -1;
448
449 poller = (IOPolledInterface *) vars->pollers->getObject(0);
450 err = poller->startIO(operation, bufferOffset, deviceOffset + vars->block0, length, completion);
451 if (err)
452 HIBLOG("IOPolledInterface::startIO[%d] 0x%x\n", 0, err);
453
454 return (err);
455}
456
457static IOReturn
458IOHibernatePollerIODone(IOPolledFileIOVars * vars)
459{
460 IOReturn err = kIOReturnError;
461 int32_t idx;
462 IOPolledInterface * poller;
463
464 while (-1 == vars->ioStatus)
465 {
466 for (idx = 0;
467 (poller = (IOPolledInterface *) vars->pollers->getObject(idx));
468 idx++)
469 {
470 err = poller->checkForWork();
471 if (err)
472 HIBLOG("IOPolledInterface::checkForWork[%d] 0x%x\n", idx, err);
473 }
474 }
475
476 if (kIOReturnSuccess != vars->ioStatus)
477 HIBLOG("IOPolledInterface::ioStatus 0x%x\n", vars->ioStatus);
478
479 return (vars->ioStatus);
480}
481
482IOReturn
483IOPolledInterface::checkAllForWork(void)
484{
485 IOReturn err = kIOReturnNotReady;
486 int32_t idx;
487 IOPolledInterface * poller;
488
489 IOHibernateVars * vars = &gIOHibernateVars;
490
491 if (!vars->fileVars || !vars->fileVars->pollers)
492 return (err);
493
494 for (idx = 0;
495 (poller = (IOPolledInterface *) vars->fileVars->pollers->getObject(idx));
496 idx++)
497 {
498 err = poller->checkForWork();
499 if (err)
500 HIBLOG("IOPolledInterface::checkAllForWork[%d] 0x%x\n", idx, err);
501 }
502
503 return (err);
504}
505
506/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
507
508struct _OpenFileContext
509{
510 OSData * extents;
511 uint64_t size;
512};
513
514static void
515file_extent_callback(void * ref, uint64_t start, uint64_t length)
516{
517 _OpenFileContext * ctx = (_OpenFileContext *) ref;
518 IOPolledFileExtent extent;
519
520 extent.start = start;
521 extent.length = length;
522
523 ctx->extents->appendBytes(&extent, sizeof(extent));
524 ctx->size += length;
525}
526
527IOReturn
528IOPolledFileOpen( const char * filename, IOBufferMemoryDescriptor * ioBuffer,
529 IOPolledFileIOVars ** fileVars, OSData ** fileExtents,
530 OSData ** imagePath)
531{
532 IOReturn err = kIOReturnError;
533 IOPolledFileIOVars * vars;
534 _OpenFileContext ctx;
535 OSData * extentsData;
536 OSNumber * num;
537 IORegistryEntry * part = 0;
538 OSDictionary * matching;
539 OSIterator * iter;
540 dev_t hibernate_image_dev;
541 uint64_t maxiobytes;
542
543 vars = &gFileVars;
544 do
545 {
546 HIBLOG("sizeof(IOHibernateImageHeader) == %ld\n", sizeof(IOHibernateImageHeader));
547 if (sizeof(IOHibernateImageHeader) != 512)
548 continue;
549
550 vars->io = false;
551 vars->buffer = (uint8_t *) ioBuffer->getBytesNoCopy();
552 vars->bufferHalf = 0;
553 vars->bufferOffset = 0;
554 vars->bufferSize = ioBuffer->getLength() >> 1;
555
556 extentsData = OSData::withCapacity(32);
557
558 ctx.extents = extentsData;
559 ctx.size = 0;
560 vars->fileRef = kern_open_file_for_direct_io(filename,
561 &file_extent_callback, &ctx,
562 &hibernate_image_dev,
563 &vars->block0,
564 &maxiobytes);
565 if (!vars->fileRef)
566 {
567 err = kIOReturnNoSpace;
568 break;
569 }
570 HIBLOG("Opened file %s, size %qd, partition base 0x%qx, maxio %qx\n", filename, ctx.size,
571 vars->block0, maxiobytes);
572 if (ctx.size < 1*1024*1024) // check against image size estimate!
573 {
574 err = kIOReturnNoSpace;
575 break;
576 }
577
578 if (maxiobytes < vars->bufferSize)
579 vars->bufferSize = maxiobytes;
580
581 vars->extentMap = (IOPolledFileExtent *) extentsData->getBytesNoCopy();
582
583 matching = IOService::serviceMatching("IOMedia");
584 num = OSNumber::withNumber(major(hibernate_image_dev), 32);
585 matching->setObject(kIOBSDMajorKey, num);
586 num->release();
587 num = OSNumber::withNumber(minor(hibernate_image_dev), 32);
588 matching->setObject(kIOBSDMinorKey, num);
589 num->release();
590 iter = IOService::getMatchingServices(matching);
591 matching->release();
592 if (iter)
593 {
594 part = (IORegistryEntry *) iter->getNextObject();
595 part->retain();
596 iter->release();
597 }
598
599 int minor, major;
600 IORegistryEntry * next;
601 IORegistryEntry * child;
602 OSData * data;
603
604 num = (OSNumber *) part->getProperty(kIOBSDMajorKey);
605 if (!num)
606 break;
607 major = num->unsigned32BitValue();
608 num = (OSNumber *) part->getProperty(kIOBSDMinorKey);
609 if (!num)
610 break;
611 minor = num->unsigned32BitValue();
612
613 hibernate_image_dev = makedev(major, minor);
614
615 vars->pollers = OSArray::withCapacity(4);
616 if (!vars->pollers)
617 break;
618
619 vars->blockSize = 512;
620 next = part;
621 do
622 {
623 IOPolledInterface * poller;
624 if ((poller = OSDynamicCast(IOPolledInterface, next->getProperty(kIOPolledInterfaceSupportKey))))
625 vars->pollers->setObject(poller);
626 if ((num = OSDynamicCast(OSNumber, next->getProperty(kIOMediaPreferredBlockSizeKey))))
627 vars->blockSize = num->unsigned32BitValue();
628 child = next;
629 }
630 while ((next = child->getParentEntry(gIOServicePlane))
631 && child->isParent(next, gIOServicePlane, true));
632
633 HIBLOG("hibernate image major %d, minor %d, blocksize %ld, pollers %d\n",
634 major, minor, vars->blockSize, vars->pollers->getCount());
635 if (vars->pollers->getCount() < kIOHibernateMinPollersNeeded)
636 continue;
637
638 err = IOHibernatePollerProbe(vars, (IOService *) part);
639 if (kIOReturnSuccess != err)
640 break;
641
642 err = IOHibernatePollerOpen(vars, kIOPolledPreflightState, ioBuffer);
643 if (kIOReturnSuccess != err)
644 break;
645
646 *fileVars = vars;
647 *fileExtents = extentsData;
648
649 // make imagePath
650 char str1[256];
651 char str2[24];
652 int len = sizeof(str1);
653
654 if ((extentsData->getLength() >= sizeof(IOPolledFileExtent))
655 && part->getPath(str1, &len, gIODTPlane))
656 {
657 // (strip the plane name)
658 char * tail = strchr(str1, ':');
659 if (!tail)
660 tail = str1 - 1;
661 data = OSData::withBytes(tail + 1, strlen(tail + 1));
662 sprintf(str2, ",%qx", vars->extentMap[0]);
663 data->appendBytes(str2, strlen(str2));
664 *imagePath = data;
665 }
666 }
667 while (false);
668
669 if (kIOReturnSuccess != err)
670 {
671 HIBLOG("error 0x%x opening hibernation file\n", err);
672 if (vars->fileRef)
673 kern_close_file_for_direct_io(vars->fileRef);
674 }
675
676 if (part)
677 part->release();
678
679 return (err);
680}
681
682IOReturn
683IOPolledFileClose( IOPolledFileIOVars * vars )
684{
685 if (vars->pollers)
686 {
687 IOHibernatePollerClose(vars, kIOPolledPostflightState);
688 vars->pollers->release();
689 }
690
691 gIOHibernateFileRef = vars->fileRef;
692
693 bzero(vars, sizeof(IOPolledFileIOVars));
694
695 return (kIOReturnSuccess);
696}
697
698static IOReturn
699IOPolledFileSeek(IOPolledFileIOVars * vars, uint64_t position)
700{
701 IOPolledFileExtent * extentMap;
702
703 extentMap = vars->extentMap;
704
705 vars->position = position;
706
707 while (position >= extentMap->length)
708 {
709 position -= extentMap->length;
710 extentMap++;
711 }
712
713 vars->currentExtent = extentMap;
714 vars->extentRemaining = extentMap->length - position;
715 vars->extentPosition = vars->position - position;
716
717 if (vars->bufferSize <= vars->extentRemaining)
718 vars->bufferLimit = vars->bufferSize;
719 else
720 vars->bufferLimit = vars->extentRemaining;
721
722 return (kIOReturnSuccess);
723}
724
725static IOReturn
726IOPolledFileWrite(IOPolledFileIOVars * vars,
727 const uint8_t * bytes, IOByteCount size,
728 hibernate_cryptvars_t * cryptvars)
729{
730 IOReturn err = kIOReturnSuccess;
731 IOByteCount copy;
732 bool flush = false;
733
734 do
735 {
736 if (!bytes && !size)
737 {
738 // seek to end of block & flush
739 size = vars->position & (vars->blockSize - 1);
740 if (size)
741 size = vars->blockSize - size;
742 flush = true;
743 // use some garbage for the fill
744 bytes = vars->buffer + vars->bufferOffset;
745 }
746
747 copy = vars->bufferLimit - vars->bufferOffset;
748 if (copy > size)
749 copy = size;
750 else
751 flush = true;
752
753 if (bytes)
754 {
755 bcopy(bytes, vars->buffer + vars->bufferHalf + vars->bufferOffset, copy);
756 bytes += copy;
757 }
758 else
759 bzero(vars->buffer + vars->bufferHalf + vars->bufferOffset, copy);
760
761 size -= copy;
762 vars->bufferOffset += copy;
763 vars->position += copy;
764
765 if (flush && vars->bufferOffset)
766 {
767 uint64_t offset = (vars->position - vars->bufferOffset
768 - vars->extentPosition + vars->currentExtent->start);
769 uint32_t length = (vars->bufferOffset);
770
771 if (cryptvars && vars->encryptStart && (vars->position > vars->encryptStart))
772 {
773 uint32_t encryptLen, encryptStart;
774 encryptLen = vars->position - vars->encryptStart;
775 if (encryptLen > length)
776 encryptLen = length;
777 encryptStart = length - encryptLen;
778
779 // encrypt the buffer
780 aes_encrypt_cbc(vars->buffer + vars->bufferHalf + encryptStart,
781 &cryptvars->aes_iv[0],
782 encryptLen / AES_BLOCK_SIZE,
783 vars->buffer + vars->bufferHalf + encryptStart,
784 &cryptvars->ctx.encrypt);
785 // save initial vector for following encrypts
786 bcopy(vars->buffer + vars->bufferHalf + encryptStart + encryptLen - AES_BLOCK_SIZE,
787 &cryptvars->aes_iv[0],
788 AES_BLOCK_SIZE);
789 }
790
791 if (vars->io)
792 {
793 err = IOHibernatePollerIODone(vars);
794 if (kIOReturnSuccess != err)
795 break;
796 }
797
798if (vars->position & (vars->blockSize - 1)) HIBLOG("misaligned file pos %qx\n", vars->position);
799//if (length != vars->bufferSize) HIBLOG("short write of %qx ends@ %qx\n", length, offset + length);
800
801 err = IOHibernatePollerIO(vars, kIOPolledWrite, vars->bufferHalf, offset, length);
802 if (kIOReturnSuccess != err)
803 break;
804 vars->io = true;
805
806 vars->extentRemaining -= vars->bufferOffset;
807 if (!vars->extentRemaining)
808 {
809 vars->currentExtent++;
810 vars->extentRemaining = vars->currentExtent->length;
811 vars->extentPosition = vars->position;
812 if (!vars->extentRemaining)
813 {
814 err = kIOReturnOverrun;
815 break;
816 }
817 }
818
819 vars->bufferHalf = vars->bufferHalf ? 0 : vars->bufferSize;
820 vars->bufferOffset = 0;
821 if (vars->bufferSize <= vars->extentRemaining)
822 vars->bufferLimit = vars->bufferSize;
823 else
824 vars->bufferLimit = vars->extentRemaining;
825
826 flush = false;
827 }
828 }
829 while (size);
830
831 return (err);
832}
833
834static IOReturn
835IOPolledFileRead(IOPolledFileIOVars * vars,
836 uint8_t * bytes, IOByteCount size,
837 hibernate_cryptvars_t * cryptvars)
838{
839 IOReturn err = kIOReturnSuccess;
840 IOByteCount copy;
841
842// bytesWritten += size;
843
844 do
845 {
846 copy = vars->bufferLimit - vars->bufferOffset;
847 if (copy > size)
848 copy = size;
849
850 if (bytes)
851 {
852 bcopy(vars->buffer + vars->bufferHalf + vars->bufferOffset, bytes, copy);
853 bytes += copy;
854 }
855 size -= copy;
856 vars->bufferOffset += copy;
857// vars->position += copy;
858
859 if (vars->bufferOffset == vars->bufferLimit)
860 {
861 if (vars->io)
862 {
863 err = IOHibernatePollerIODone(vars);
864 if (kIOReturnSuccess != err)
865 break;
866 }
867 else
868 cryptvars = 0;
869
870if (vars->position & (vars->blockSize - 1)) HIBLOG("misaligned file pos %qx\n", vars->position);
871
872 vars->position += vars->lastRead;
873 vars->extentRemaining -= vars->lastRead;
874 vars->bufferLimit = vars->lastRead;
875
876 if (!vars->extentRemaining)
877 {
878 vars->currentExtent++;
879 vars->extentRemaining = vars->currentExtent->length;
880 vars->extentPosition = vars->position;
881 if (!vars->extentRemaining)
882 {
883 err = kIOReturnOverrun;
884 break;
885 }
886 }
887
888 if (vars->extentRemaining <= vars->bufferSize)
889 vars->lastRead = vars->extentRemaining;
890 else
891 vars->lastRead = vars->bufferSize;
892
893 uint64_t offset = (vars->position
894 - vars->extentPosition + vars->currentExtent->start);
895 uint64_t length = (vars->lastRead);
896
897//if (length != vars->bufferSize) HIBLOG("short read of %qx ends@ %qx\n", length, offset + length);
898
899 err = IOHibernatePollerIO(vars, kIOPolledRead, vars->bufferHalf, offset, length);
900 if (kIOReturnSuccess != err)
901 break;
902 vars->io = true;
903
904 vars->bufferHalf = vars->bufferHalf ? 0 : vars->bufferSize;
905 vars->bufferOffset = 0;
906
907 if (cryptvars)
908 {
909 uint8_t thisVector[AES_BLOCK_SIZE];
910 // save initial vector for following decrypts
911 bcopy(&cryptvars->aes_iv[0], &thisVector[0], AES_BLOCK_SIZE);
912 bcopy(vars->buffer + vars->bufferHalf + vars->lastRead - AES_BLOCK_SIZE,
913 &cryptvars->aes_iv[0], AES_BLOCK_SIZE);
914 // decrypt the buffer
915 aes_decrypt_cbc(vars->buffer + vars->bufferHalf,
916 &thisVector[0],
917 vars->lastRead / AES_BLOCK_SIZE,
918 vars->buffer + vars->bufferHalf,
919 &cryptvars->ctx.decrypt);
920 }
921 }
922 }
923 while (size);
924
925 return (err);
926}
927
928/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
929
930IOReturn
931IOHibernateSystemSleep(void)
932{
933 IOReturn err;
934 OSData * data;
935 OSObject * obj;
936 OSString * str;
937 OSNumber * num;
938
939 IOHibernateVars * vars = &gIOHibernateVars;
940
941 if (vars->fileVars && vars->fileVars->fileRef)
942 // already on the way down
943 return (kIOReturnSuccess);
944
945 gIOHibernateState = kIOHibernateStateInactive;
946
947 if ((obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateModeKey)))
948 {
949 if ((num = OSDynamicCast(OSNumber, obj)))
950 gIOHibernateMode = num->unsigned32BitValue();
951 if (kIOHibernateModeSleep & gIOHibernateMode)
952 // default to discard clean for safe sleep
953 gIOHibernateMode ^= (kIOHibernateModeDiscardCleanInactive
954 | kIOHibernateModeDiscardCleanActive);
955
956 obj->release();
957 }
958 if ((obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateFreeRatioKey)))
959 {
960 if ((num = OSDynamicCast(OSNumber, obj)))
961 gIOHibernateFreeRatio = num->unsigned32BitValue();
962 obj->release();
963 }
964 if ((obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateFreeTimeKey)))
965 {
966 if ((num = OSDynamicCast(OSNumber, obj)))
967 gIOHibernateFreeTime = num->unsigned32BitValue();
968 obj->release();
969 }
970 if ((obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateFileKey)))
971 {
972 if ((str = OSDynamicCast(OSString, obj)))
973 strcpy(gIOHibernateFilename, str->getCStringNoCopy());
974 obj->release();
975 }
976
977 if (!gIOHibernateMode || !gIOHibernateFilename[0])
978 return (kIOReturnUnsupported);
979
980 HIBLOG("hibernate image path: %s\n", gIOHibernateFilename);
981
982 do
983 {
984 vars->srcBuffer = IOBufferMemoryDescriptor::withOptions(0, 4 * page_size, page_size);
985 vars->ioBuffer = IOBufferMemoryDescriptor::withOptions(0, 2 * kDefaultIOSize, page_size);
986
987 if (!vars->srcBuffer || !vars->ioBuffer)
988 {
989 err = kIOReturnNoMemory;
990 break;
991 }
992
993 err = IOPolledFileOpen(gIOHibernateFilename, vars->ioBuffer,
994 &vars->fileVars, &vars->fileExtents, &data);
995 if (KERN_SUCCESS != err)
996 {
997 HIBLOG("IOPolledFileOpen(%x)\n", err);
998 break;
999 }
1000 if (vars->fileVars->fileRef)
1001 {
1002 // invalidate the image file
1003 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderInvalidSignature;
1004 int err = kern_write_file(vars->fileVars->fileRef, 0,
1005 (caddr_t) gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
1006 if (KERN_SUCCESS != err)
1007 HIBLOG("kern_write_file(%d)\n", err);
1008 }
1009
1010 bzero(gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
1011
1012 boolean_t encryptedswap;
1013 err = hibernate_setup(gIOHibernateCurrentHeader,
1014 gIOHibernateFreeRatio, gIOHibernateFreeTime,
1015 &vars->page_list, &vars->page_list_wired, &encryptedswap);
1016 if (KERN_SUCCESS != err)
1017 {
1018 HIBLOG("hibernate_setup(%d)\n", err);
1019 break;
1020 }
1021
1022 if (encryptedswap)
1023 gIOHibernateMode ^= kIOHibernateModeEncrypt;
1024
1025 vars->videoAllocSize = kVideoMapSize;
1026 if (KERN_SUCCESS != kmem_alloc_pageable(kernel_map, &vars->videoMapping, vars->videoAllocSize))
1027 vars->videoMapping = 0;
1028
1029 // generate crypt keys
1030 for (uint32_t i = 0; i < sizeof(vars->wiredCryptKey); i++)
1031 vars->wiredCryptKey[i] = random();
1032 for (uint32_t i = 0; i < sizeof(vars->cryptKey); i++)
1033 vars->cryptKey[i] = random();
1034
1035 // set nvram
1036
1037 IORegistryEntry * regEntry;
1038 if (!gIOOptionsEntry)
1039 {
1040 regEntry = IORegistryEntry::fromPath("/options", gIODTPlane);
1041 gIOOptionsEntry = OSDynamicCast(IODTNVRAM, regEntry);
1042 if (regEntry && !gIOOptionsEntry)
1043 regEntry->release();
1044 }
1045 if (!gIOChosenEntry)
1046 gIOChosenEntry = IORegistryEntry::fromPath("/chosen", gIODTPlane);
1047
1048 if (gIOOptionsEntry)
1049 {
1050 const OSSymbol * sym;
1051 size_t len;
1052 char valueString[16];
1053
1054 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKey);
1055 if (sym)
1056 {
1057 gIOOptionsEntry->setProperty(sym, data);
1058 sym->release();
1059 }
1060 data->release();
1061
1062 vars->saveBootDevice = gIOOptionsEntry->copyProperty(kIOSelectedBootDeviceKey);
1063 if (gIOChosenEntry)
1064 {
1065 OSData * bootDevice = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOBootPathKey));
1066 if (bootDevice)
1067 {
1068 sym = OSSymbol::withCStringNoCopy(kIOSelectedBootDeviceKey);
1069 OSString * str2 = OSString::withCStringNoCopy((const char *) bootDevice->getBytesNoCopy());
1070 if (sym && str2)
1071 gIOOptionsEntry->setProperty(sym, str2);
1072 if (sym)
1073 sym->release();
1074 if (str2)
1075 str2->release();
1076 }
1077
1078 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMemorySignatureKey));
1079 if (data)
1080 {
1081 vars->haveFastBoot = true;
1082
1083 len = sprintf(valueString, "0x%lx", *((UInt32 *)data->getBytesNoCopy()));
1084 data = OSData::withBytes(valueString, len + 1);
1085 sym = OSSymbol::withCStringNoCopy(kIOHibernateMemorySignatureEnvKey);
1086 if (sym && data)
1087 gIOOptionsEntry->setProperty(sym, data);
1088 if (sym)
1089 sym->release();
1090 if (data)
1091 data->release();
1092 }
1093 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMachineSignatureKey));
1094 if (data)
1095 gIOHibernateCurrentHeader->machineSignature = *((UInt32 *)data->getBytesNoCopy());
1096 }
1097
1098 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1099 {
1100 data = OSData::withBytes(&vars->wiredCryptKey[0], sizeof(vars->wiredCryptKey));
1101 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKeyKey);
1102 if (sym && data)
1103 gIOOptionsEntry->setProperty(sym, data);
1104 if (sym)
1105 sym->release();
1106 if (data)
1107 data->release();
1108 if (gIOHibernateBootSignature[0])
1109 {
1110 data = OSData::withCapacity(16);
1111 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootSignatureKey);
1112 if (sym && data)
1113 {
1114 char c;
1115 uint8_t value;
1116 for (uint32_t i = 0; (c = gIOHibernateBootSignature[i]); i++)
1117 {
1118 if (c >= 'a')
1119 c -= 'a' - 10;
1120 else if (c >= 'A')
1121 c -= 'A' - 10;
1122 else if (c >= '0')
1123 c -= '0';
1124 else
1125 continue;
1126 value = (value << 4) | c;
1127 if (i & 1)
1128 data->appendBytes(&value, sizeof(value));
1129 }
1130 gIOOptionsEntry->setProperty(sym, data);
1131 }
1132 if (sym)
1133 sym->release();
1134 if (data)
1135 data->release();
1136 }
1137 }
1138
1139 if (!vars->haveFastBoot)
1140 {
1141 // set boot volume to zero
1142 IODTPlatformExpert * platform = OSDynamicCast(IODTPlatformExpert, IOService::getPlatform());
1143 if (platform && (kIOReturnSuccess == platform->readXPRAM(kXPRamAudioVolume,
1144 &vars->saveBootAudioVolume, sizeof(vars->saveBootAudioVolume))))
1145 {
1146 uint8_t newVolume;
1147 newVolume = vars->saveBootAudioVolume & 0xf8;
1148 platform->writeXPRAM(kXPRamAudioVolume,
1149 &newVolume, sizeof(newVolume));
1150 }
1151 }
1152 }
1153 // --
1154
1155 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderSignature;
1156 gIOHibernateState = kIOHibernateStateHibernating;
1157 }
1158 while (false);
1159
1160 return (err);
1161}
1162
1163/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1164
1165IOReturn
1166IOHibernateSystemHasSlept(void)
1167{
1168 IOHibernateVars * vars = &gIOHibernateVars;
1169
1170 if ((vars->previewData = OSDynamicCast(OSData,
1171 IOService::getPMRootDomain()->getProperty(kIOHibernatePreviewBufferKey))))
1172 {
1173 vars->previewBuffer = IOMemoryDescriptor::withAddress(
1174 (void *) vars->previewData->getBytesNoCopy(),
1175 vars->previewData->getLength(),
1176 kIODirectionInOut);
1177
1178 if (vars->previewBuffer && (kIOReturnSuccess != vars->previewBuffer->prepare()))
1179 {
1180 vars->previewBuffer->release();
1181 vars->previewBuffer = 0;
1182 }
1183 if (!vars->previewBuffer)
1184 vars->previewData = 0;
1185 }
1186 if (gIOOptionsEntry)
1187 gIOOptionsEntry->sync();
1188
1189 return (kIOReturnSuccess);
1190}
1191
1192/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1193
1194IOReturn
1195IOHibernateSystemWake(void)
1196{
1197 IOHibernateVars * vars = &gIOHibernateVars;
1198
1199 hibernate_teardown(vars->page_list, vars->page_list_wired);
1200
1201 if (vars->videoMapping)
1202 {
1203 if (vars->videoMapSize)
1204 // remove mappings
1205 IOUnmapPages(kernel_map, vars->videoMapping, vars->videoMapSize);
1206 if (vars->videoAllocSize)
1207 // dealloc range
1208 kmem_free(kernel_map, trunc_page_32(vars->videoMapping), vars->videoAllocSize);
1209 }
1210
1211 if (vars->previewBuffer)
1212 {
1213 vars->previewBuffer->release();
1214 vars->previewBuffer = 0;
1215 }
1216
1217 if (vars->fileVars)
1218 {
1219 IOPolledFileClose(vars->fileVars);
1220 }
1221
1222 // invalidate nvram properties - (gIOOptionsEntry != 0) => nvram was touched
1223
1224 OSData * data = OSData::withCapacity(4);
1225 if (gIOOptionsEntry && data)
1226 {
1227 const OSSymbol * sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKey);
1228 if (sym)
1229 {
1230 gIOOptionsEntry->setProperty(sym, data);
1231 sym->release();
1232 }
1233 sym = OSSymbol::withCStringNoCopy(kIOSelectedBootDeviceKey);
1234 if (sym)
1235 {
1236 if (vars->saveBootDevice)
1237 {
1238 gIOOptionsEntry->setProperty(sym, vars->saveBootDevice);
1239 vars->saveBootDevice->release();
1240 }
1241 sym->release();
1242 }
1243 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKeyKey);
1244 if (sym)
1245 {
1246 gIOOptionsEntry->setProperty(sym, data);
1247 sym->release();
1248 }
1249 sym = OSSymbol::withCStringNoCopy(kIOHibernateMemorySignatureEnvKey);
1250 if (sym)
1251 {
1252 gIOOptionsEntry->removeProperty(sym);
1253 sym->release();
1254 }
1255 }
1256 if (data)
1257 data->release();
1258
1259 if (gIOOptionsEntry)
1260 {
1261 if (!vars->haveFastBoot)
1262 {
1263 // reset boot audio volume
1264 IODTPlatformExpert * platform = OSDynamicCast(IODTPlatformExpert, IOService::getPlatform());
1265 if (platform)
1266 platform->writeXPRAM(kXPRamAudioVolume,
1267 &vars->saveBootAudioVolume, sizeof(vars->saveBootAudioVolume));
1268 }
1269
1270 // sync now to hardware if the booter has not
1271 if (kIOHibernateStateInactive == gIOHibernateState)
1272 gIOOptionsEntry->sync();
1273 else
1274 // just sync the variables in case a later panic syncs nvram (it won't sync variables)
1275 gIOOptionsEntry->syncOFVariables();
1276 }
1277
1278 if (vars->srcBuffer)
1279 vars->srcBuffer->release();
1280 if (vars->ioBuffer)
1281 vars->ioBuffer->release();
1282 if (vars->fileExtents)
1283 vars->fileExtents->release();
1284
1285 bzero(vars, sizeof(*vars));
1286
1287// gIOHibernateState = kIOHibernateStateInactive; // leave it for post wake code to see
1288
1289 return (kIOReturnSuccess);
1290}
1291
1292IOReturn
1293IOHibernateSystemPostWake(void)
1294{
1295 if (gIOHibernateFileRef)
1296 {
1297 kern_close_file_for_direct_io(gIOHibernateFileRef);
1298 gIOHibernateFileRef = 0;
1299 }
1300 return (kIOReturnSuccess);
1301}
1302
1303/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1304
1305void
1306IOHibernateSystemInit(IOPMrootDomain * rootDomain)
1307{
1308 OSData * data = OSData::withBytesNoCopy(&gIOHibernateState, sizeof(gIOHibernateState));
1309 if (data)
1310 {
1311 rootDomain->setProperty(kIOHibernateStateKey, data);
1312 data->release();
1313 }
1314
1315 if (PE_parse_boot_arg("hfile", gIOHibernateFilename))
1316 gIOHibernateMode = kIOHibernateModeOn;
1317 else
1318 gIOHibernateFilename[0] = 0;
1319
1320 static SYSCTL_STRING(_kern, OID_AUTO, hibernatefile,
1321 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1322 gIOHibernateFilename, sizeof(gIOHibernateFilename), "");
1323 sysctl_register_oid(&sysctl__kern_hibernatefile);
1324
1325 static SYSCTL_STRING(_kern, OID_AUTO, bootsignature,
1326 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1327 gIOHibernateBootSignature, sizeof(gIOHibernateBootSignature), "");
1328 sysctl_register_oid(&sysctl__kern_bootsignature);
1329
1330 static SYSCTL_UINT(_kern, OID_AUTO, hibernatemode,
1331 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1332 &gIOHibernateMode, 0, "");
1333 sysctl_register_oid(&sysctl__kern_hibernatemode);
1334}
1335
1336/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1337
1338static void
1339hibernate_setup_for_wake(void)
1340{
1341#if __ppc__
1342 // go slow (state needed for wake)
1343 ml_set_processor_speed(1);
1344#endif
1345}
1346
1347/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1348
1349extern "C" boolean_t
1350hibernate_write_image(void)
1351{
1352 IOHibernateImageHeader * header = gIOHibernateCurrentHeader;
1353 IOHibernateVars * vars = &gIOHibernateVars;
1354 IOPolledFileExtent * fileExtents;
1355
1356 uint32_t pageCount, pagesDone;
1357 IOReturn err;
1358 vm_offset_t ppnum;
1359 IOItemCount page, count;
1360 uint8_t * src;
1361 uint8_t * data;
1362 IOByteCount pageCompressedSize;
1363 uint64_t compressedSize, uncompressedSize;
1364 uint64_t image1Size = 0;
1365 uint32_t bitmap_size;
1366 bool iterDone, pollerOpen, needEncryptStart;
1367 uint32_t restore1Sum, sum, sum1, sum2;
1368 uint32_t tag;
1369 uint32_t pageType;
1370 uint32_t pageAndCount[2];
1371
1372 AbsoluteTime startTime, endTime;
1373 AbsoluteTime allTime, compTime, decoTime;
1374 uint64_t nsec;
1375 uint32_t lastProgressStamp = 0;
1376 uint32_t progressStamp;
1377
1378 hibernate_cryptvars_t _cryptvars;
1379 hibernate_cryptvars_t * cryptvars = 0;
1380
1381 if (!vars->fileVars || !vars->fileVars->pollers || !vars->fileExtents)
1382 return (false /* sleep */ );
1383
1384 restore1Sum = sum1 = sum2 = 0;
1385
1386 // encryption data. "iv" is the "initial vector".
1387 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1388 {
1389 static const unsigned char first_iv[AES_BLOCK_SIZE]
1390 = { 0xa3, 0x63, 0x65, 0xa9, 0x0b, 0x71, 0x7b, 0x1c,
1391 0xdf, 0x9e, 0x5f, 0x32, 0xd7, 0x61, 0x63, 0xda };
1392
1393 cryptvars = &gIOHibernateCryptWakeContext;
1394 bzero(cryptvars, sizeof(hibernate_cryptvars_t));
1395 aes_encrypt_key(vars->cryptKey,
1396 kIOHibernateAESKeySize,
1397 &cryptvars->ctx.encrypt);
1398 aes_decrypt_key(vars->cryptKey,
1399 kIOHibernateAESKeySize,
1400 &cryptvars->ctx.decrypt);
1401
1402 cryptvars = &_cryptvars;
1403 bzero(cryptvars, sizeof(hibernate_cryptvars_t));
1404 aes_encrypt_key(vars->wiredCryptKey,
1405 kIOHibernateAESKeySize,
1406 &cryptvars->ctx.encrypt);
1407
1408 bcopy(&first_iv[0], &cryptvars->aes_iv[0], AES_BLOCK_SIZE);
1409 bzero(&vars->wiredCryptKey[0], sizeof(vars->wiredCryptKey));
1410 bzero(&vars->cryptKey[0], sizeof(vars->cryptKey));
1411 bzero(gIOHibernateCryptWakeVars, sizeof(hibernate_cryptwakevars_t));
1412 }
1413
1414 hibernate_setup_for_wake();
1415
1416 hibernate_page_list_setall(vars->page_list,
1417 vars->page_list_wired,
1418 &pageCount);
1419
1420 HIBLOG("hibernate_page_list_setall found pageCount %d\n", pageCount);
1421
1422 fileExtents = (IOPolledFileExtent *) vars->fileExtents->getBytesNoCopy();
1423
1424#if 0
1425 count = vars->fileExtents->getLength() / sizeof(IOPolledFileExtent);
1426 for (page = 0; page < count; page++)
1427 {
1428 HIBLOG("fileExtents[%d] %qx, %qx (%qx)\n", page,
1429 fileExtents[page].start, fileExtents[page].length,
1430 fileExtents[page].start + fileExtents[page].length);
1431 }
1432#endif
1433
1434 needEncryptStart = (0 != (kIOHibernateModeEncrypt & gIOHibernateMode));
1435
1436 AbsoluteTime_to_scalar(&compTime) = 0;
1437 AbsoluteTime_to_scalar(&decoTime) = 0;
1438
1439 clock_get_uptime(&allTime);
1440
1441 do
1442 {
1443 compressedSize = 0;
1444 uncompressedSize = 0;
1445 iterDone = false;
1446 pageType = 0; // wired pages first
1447
1448 IOPolledFileSeek(vars->fileVars, sizeof(IOHibernateImageHeader));
1449
1450 HIBLOG("IOHibernatePollerOpen, ml_get_interrupts_enabled %d\n",
1451 ml_get_interrupts_enabled());
1452 err = IOHibernatePollerOpen(vars->fileVars, kIOPolledBeforeSleepState, vars->ioBuffer);
1453 HIBLOG("IOHibernatePollerOpen(%x)\n", err);
1454 pollerOpen = (kIOReturnSuccess == err);
1455 if (!pollerOpen)
1456 break;
1457
1458 // copy file block extent list if larger than header
1459
1460 count = vars->fileExtents->getLength();
1461 if (count > sizeof(header->fileExtentMap))
1462 {
1463 count -= sizeof(header->fileExtentMap);
1464 err = IOPolledFileWrite(vars->fileVars,
1465 ((uint8_t *) &fileExtents[0]) + sizeof(header->fileExtentMap), count, cryptvars);
1466 if (kIOReturnSuccess != err)
1467 break;
1468 }
1469
1470 // copy out restore1 code
1471
1472 page = atop_32(sectHIBB);
1473 count = atop_32(round_page(sectHIBB + sectSizeHIB)) - page;
1474 header->restore1CodePage = page;
1475 header->restore1PageCount = count;
1476 header->restore1CodeOffset = ((uint32_t) &hibernate_machine_entrypoint) - sectHIBB;
1477 header->restore1StackOffset = ((uint32_t) &gIOHibernateRestoreStackEnd[0]) - 64 - sectHIBB;
1478
1479 // sum __HIB sect, with zeros for the stack
1480 src = (uint8_t *) trunc_page(sectHIBB);
1481 for (page = 0; page < count; page++)
1482 {
1483 if ((src < &gIOHibernateRestoreStack[0]) || (src >= &gIOHibernateRestoreStackEnd[0]))
1484 restore1Sum += hibernate_sum(src, page_size);
1485 else
1486 restore1Sum += 0x10000001;
1487 src += page_size;
1488 }
1489 sum1 = restore1Sum;
1490
1491 // write the __HIB sect, with zeros for the stack
1492
1493 src = (uint8_t *) trunc_page(sectHIBB);
1494 count = ((uint32_t) &gIOHibernateRestoreStack[0]) - trunc_page(sectHIBB);
1495 if (count)
1496 {
1497 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1498 if (kIOReturnSuccess != err)
1499 break;
1500 }
1501 err = IOPolledFileWrite(vars->fileVars,
1502 (uint8_t *) 0,
1503 &gIOHibernateRestoreStackEnd[0] - &gIOHibernateRestoreStack[0],
1504 cryptvars);
1505 if (kIOReturnSuccess != err)
1506 break;
1507 src = &gIOHibernateRestoreStackEnd[0];
1508 count = round_page(sectHIBB + sectSizeHIB) - ((uint32_t) src);
1509 if (count)
1510 {
1511 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1512 if (kIOReturnSuccess != err)
1513 break;
1514 }
1515
1516 // write the preview buffer
1517
1518 addr64_t phys64;
1519 IOByteCount segLen;
1520
1521 if (vars->previewData)
1522 {
1523 ppnum = 0;
1524 count = 0;
1525 do
1526 {
1527 phys64 = vars->previewBuffer->getPhysicalSegment64(count, &segLen);
1528 pageAndCount[0] = atop_64(phys64);
1529 pageAndCount[1] = atop_32(segLen);
1530 err = IOPolledFileWrite(vars->fileVars,
1531 (const uint8_t *) &pageAndCount, sizeof(pageAndCount),
1532 cryptvars);
1533 if (kIOReturnSuccess != err)
1534 break;
1535 count += segLen;
1536 ppnum += sizeof(pageAndCount);
1537 }
1538 while (phys64);
1539 if (kIOReturnSuccess != err)
1540 break;
1541
1542 src = (uint8_t *) vars->previewData->getBytesNoCopy();
1543 count = vars->previewData->getLength();
1544
1545 header->previewPageListSize = ppnum;
1546 header->previewSize = count + ppnum;
1547
1548 for (page = 0; page < count; page += page_size)
1549 sum1 += hibernate_sum(src + page, page_size);
1550
1551 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1552 if (kIOReturnSuccess != err)
1553 break;
1554 }
1555
1556 // mark areas for no save
1557
1558 for (count = 0;
1559 (phys64 = vars->ioBuffer->getPhysicalSegment64(count, &segLen));
1560 count += segLen)
1561 {
1562 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1563 atop_64(phys64), atop_32(segLen),
1564 kIOHibernatePageStateFree);
1565 pageCount -= atop_32(segLen);
1566 }
1567
1568 for (count = 0;
1569 (phys64 = vars->srcBuffer->getPhysicalSegment64(count, &segLen));
1570 count += segLen)
1571 {
1572 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1573 atop_64(phys64), atop_32(segLen),
1574 kIOHibernatePageStateFree);
1575 pageCount -= atop_32(segLen);
1576 }
1577
1578 // copy out bitmap of pages available for trashing during restore
1579
1580 bitmap_size = vars->page_list_wired->list_size;
1581 src = (uint8_t *) vars->page_list_wired;
1582 err = IOPolledFileWrite(vars->fileVars, src, bitmap_size, cryptvars);
1583 if (kIOReturnSuccess != err)
1584 break;
1585
1586 // mark more areas for no save, but these are not available
1587 // for trashing during restore
1588
1589#if !__i386__
1590 page = atop_32(sectHIBB);
1591 count = atop_32(round_page(sectHIBB + sectSizeHIB)) - page;
1592#else
1593 // XXX
1594 page = atop_32(sectHIBB & 0x3FFFFFFF);
1595 count = atop_32(round_page((sectHIBB + sectSizeHIB) & 0x3FFFFFFF)) - page;
1596#endif
1597 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1598 page, count,
1599 kIOHibernatePageStateFree);
1600 pageCount -= count;
1601
1602
1603
1604 if (vars->previewBuffer) for (count = 0;
1605 (phys64 = vars->previewBuffer->getPhysicalSegment64(count, &segLen));
1606 count += segLen)
1607 {
1608 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1609 atop_64(phys64), atop_32(segLen),
1610 kIOHibernatePageStateFree);
1611 pageCount -= atop_32(segLen);
1612 }
1613
1614 src = (uint8_t *) vars->srcBuffer->getBytesNoCopy();
1615
1616 void * iter = 0;
1617 pagesDone = 0;
1618
1619 HIBLOG("writing %d pages\n", pageCount);
1620
1621 do
1622 {
1623 count = hibernate_page_list_iterate(pageType ? vars->page_list : vars->page_list_wired,
1624 &iter, &ppnum);
1625// kprintf("[%d](%x : %x)\n", pageType, ppnum, count);
1626
1627 iterDone = !count;
1628
1629 pageAndCount[0] = ppnum;
1630 pageAndCount[1] = count;
1631 err = IOPolledFileWrite(vars->fileVars,
1632 (const uint8_t *) &pageAndCount, sizeof(pageAndCount),
1633 cryptvars);
1634 if (kIOReturnSuccess != err)
1635 break;
1636
1637 for (page = 0; page < count; page++)
1638 {
1639 err = IOMemoryDescriptorWriteFromPhysical(vars->srcBuffer, 0, ptoa_64(ppnum), page_size);
1640 if (err)
1641 {
1642 HIBLOG("IOMemoryDescriptorWriteFromPhysical %d [%d] %x\n", __LINE__, ppnum, err);
1643 break;
1644 }
1645
1646 sum = hibernate_sum(src, page_size);
1647
1648 clock_get_uptime(&startTime);
1649
1650 pageCompressedSize = WKdm_compress ((WK_word*) src, (WK_word*) (src + page_size), PAGE_SIZE_IN_WORDS);
1651
1652 clock_get_uptime(&endTime);
1653 ADD_ABSOLUTETIME(&compTime, &endTime);
1654 SUB_ABSOLUTETIME(&compTime, &startTime);
1655
1656 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1657 pageCompressedSize = (pageCompressedSize + AES_BLOCK_SIZE - 1) & ~(AES_BLOCK_SIZE - 1);
1658
1659 if (pageCompressedSize > page_size)
1660 {
1661// HIBLOG("------------lose: %d\n", pageCompressedSize);
1662 pageCompressedSize = page_size;
1663 }
1664
1665 if (pageCompressedSize != page_size)
1666 data = (src + page_size);
1667 else
1668 data = src;
1669
1670 tag = pageCompressedSize | kIOHibernateTagSignature;
1671
1672 if (pageType)
1673 sum2 += sum;
1674 else
1675 sum1 += sum;
1676
1677 if (needEncryptStart && (ppnum >= atop_32(sectDATAB)))
1678 {
1679 // start encrypting partway into the data about to be written
1680 vars->fileVars->encryptStart = (vars->fileVars->position + AES_BLOCK_SIZE - 1)
1681 & ~(AES_BLOCK_SIZE - 1);
1682 needEncryptStart = false;
1683 }
1684
1685 err = IOPolledFileWrite(vars->fileVars, (const uint8_t *) &tag, sizeof(tag), cryptvars);
1686 if (kIOReturnSuccess != err)
1687 break;
1688
1689 err = IOPolledFileWrite(vars->fileVars, data, (pageCompressedSize + 3) & ~3, cryptvars);
1690 if (kIOReturnSuccess != err)
1691 break;
1692
1693 compressedSize += pageCompressedSize;
1694 if (pageCompressedSize)
1695 uncompressedSize += page_size;
1696 ppnum++;
1697 pagesDone++;
1698
1699 if (0 == (8191 & pagesDone))
1700 {
1701 clock_get_uptime(&endTime);
1702 SUB_ABSOLUTETIME(&endTime, &allTime);
1703 absolutetime_to_nanoseconds(endTime, &nsec);
1704 progressStamp = nsec / 750000000ULL;
1705 if (progressStamp != lastProgressStamp)
1706 {
1707 lastProgressStamp = progressStamp;
1708 HIBPRINT("pages %d (%d%%)\n", pagesDone, (100 * pagesDone) / pageCount);
1709 }
1710 }
1711 }
1712 if (kIOReturnSuccess != err)
1713 break;
1714 if (iterDone && !pageType)
1715 {
1716 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
1717 if (kIOReturnSuccess != err)
1718 break;
1719
1720 iterDone = false;
1721 pageType = 1;
1722 iter = 0;
1723 image1Size = vars->fileVars->position;
1724 if (cryptvars)
1725 {
1726 bcopy(&cryptvars->aes_iv[0],
1727 &gIOHibernateCryptWakeContext.aes_iv[0],
1728 sizeof(cryptvars->aes_iv));
1729 cryptvars = &gIOHibernateCryptWakeContext;
1730 }
1731 HIBLOG("image1Size %qd\n", image1Size);
1732 }
1733 }
1734 while (!iterDone);
1735 if (kIOReturnSuccess != err)
1736 break;
1737 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
1738 if (kIOReturnSuccess != err)
1739 break;
1740
1741 // Header:
1742
1743 header->imageSize = vars->fileVars->position;
1744 header->image1Size = image1Size;
1745 header->bitmapSize = bitmap_size;
1746 header->pageCount = pageCount;
1747 header->encryptStart = vars->fileVars->encryptStart;
1748
1749 header->restore1Sum = restore1Sum;
1750 header->image1Sum = sum1;
1751 header->image2Sum = sum2;
1752
1753 count = vars->fileExtents->getLength();
1754 if (count > sizeof(header->fileExtentMap))
1755 {
1756 header->fileExtentMapSize = count;
1757 count = sizeof(header->fileExtentMap);
1758 }
1759 else
1760 header->fileExtentMapSize = sizeof(header->fileExtentMap);
1761 bcopy(&fileExtents[0], &header->fileExtentMap[0], count);
1762
1763 IOPolledFileSeek(vars->fileVars, 0);
1764 err = IOPolledFileWrite(vars->fileVars,
1765 (uint8_t *) header, sizeof(IOHibernateImageHeader),
1766 cryptvars);
1767 if (kIOReturnSuccess != err)
1768 break;
1769 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
1770 if (kIOReturnSuccess != err)
1771 break;
1772 err = IOHibernatePollerIODone(vars->fileVars);
1773 if (kIOReturnSuccess != err)
1774 break;
1775 }
1776 while (false);
1777
1778 clock_get_uptime(&endTime);
1779 SUB_ABSOLUTETIME(&endTime, &allTime);
1780 absolutetime_to_nanoseconds(endTime, &nsec);
1781 HIBLOG("all time: %qd ms, ",
1782 nsec / 1000000ULL);
1783
1784 absolutetime_to_nanoseconds(compTime, &nsec);
1785 HIBLOG("comp time: %qd ms, ",
1786 nsec / 1000000ULL);
1787
1788 absolutetime_to_nanoseconds(decoTime, &nsec);
1789 HIBLOG("deco time: %qd ms, ",
1790 nsec / 1000000ULL);
1791
1792 HIBLOG("\nimage %qd, uncompressed %qd (%d), compressed %qd (%d%%), sum1 %x, sum2 %x\n",
1793 header->imageSize,
1794 uncompressedSize, atop_32(uncompressedSize), compressedSize,
1795 (int) ((compressedSize * 100ULL) / uncompressedSize),
1796 sum1, sum2);
1797
1798 if (pollerOpen)
1799 IOHibernatePollerClose(vars->fileVars, kIOPolledBeforeSleepState);
1800
1801 HIBLOG("hibernate_write_image done(%x)\n", err);
1802
1803 // should we come back via regular wake, set the state in memory.
1804 gIOHibernateState = kIOHibernateStateInactive;
1805
1806 if ((kIOReturnSuccess == err) && !(kIOHibernateModeSleep & gIOHibernateMode))
1807 return (true /* power down */ );
1808 else
1809 return (false /* sleep */ );
1810}
1811
1812/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1813
1814DECLARE_IOHIBERNATEPROGRESSALPHA
1815
1816static void
1817ProgressUpdate(hibernate_graphics_t * display, uint8_t * screen, int32_t firstBlob, int32_t select)
1818{
1819 uint32_t rowBytes, pixelShift;
1820 uint32_t x, y;
1821 int32_t blob, lastBlob;
1822 uint32_t alpha, in, color, result;
1823 uint8_t * out;
1824 uint32_t saveindex[kIOHibernateProgressCount] = { 0 };
1825
1826 pixelShift = display->depth >> 4;
1827 if (pixelShift < 1)
1828 return;
1829
1830 rowBytes = display->rowBytes;
1831
1832 screen += ((display->width
1833 - kIOHibernateProgressCount * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << (pixelShift - 1))
1834 + (display->height - kIOHibernateProgressOriginY - kIOHibernateProgressHeight) * rowBytes;
1835
1836 lastBlob = (select < kIOHibernateProgressCount) ? select : (kIOHibernateProgressCount - 1);
1837
1838 screen += (firstBlob * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << pixelShift;
1839
1840 for (y = 0; y < kIOHibernateProgressHeight; y++)
1841 {
1842 out = screen + y * rowBytes;
1843 for (blob = firstBlob; blob <= lastBlob; blob++)
1844 {
1845 color = (blob < select) ? kIOHibernateProgressLightGray : kIOHibernateProgressMidGray;
1846 for (x = 0; x < kIOHibernateProgressWidth; x++)
1847 {
1848 alpha = gIOHibernateProgressAlpha[y][x];
1849 result = color;
1850 if (alpha)
1851 {
1852 if (0xff != alpha)
1853 {
1854 in = display->progressSaveUnder[blob][saveindex[blob]++];
1855 result = ((255 - alpha) * in + alpha * result + 0xff) / 255;
1856 }
1857 if (1 == pixelShift)
1858 {
1859 result >>= 3;
1860 *((uint16_t *)out) = (result << 10) | (result << 5) | result; // 16
1861 }
1862 else
1863 *((uint32_t *)out) = (result << 16) | (result << 8) | result; // 32
1864 }
1865 out += (1 << pixelShift);
1866 }
1867 out += (kIOHibernateProgressSpacing << pixelShift);
1868 }
1869 }
1870}
1871
1872/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1873
1874extern "C" void
1875hibernate_machine_init(void)
1876{
1877 IOReturn err;
1878 uint32_t sum;
1879 uint32_t pagesDone;
1880 AbsoluteTime allTime, endTime;
1881 uint64_t nsec;
1882 uint32_t lastProgressStamp = 0;
1883 uint32_t progressStamp;
1884 uint64_t progressZeroPosition = 0;
1885 uint32_t blob, lastBlob = (uint32_t) -1L;
1886 hibernate_cryptvars_t * cryptvars = 0;
1887
1888 IOHibernateVars * vars = &gIOHibernateVars;
1889
1890 if (!vars->fileVars || !vars->fileVars->pollers || !vars->fileExtents)
1891 return;
1892
1893 if ((kIOHibernateModeDiscardCleanActive | kIOHibernateModeDiscardCleanInactive) & gIOHibernateMode)
1894 hibernate_page_list_discard(vars->page_list);
1895
1896
1897 sum = gIOHibernateCurrentHeader->actualImage1Sum;
1898 pagesDone = gIOHibernateCurrentHeader->actualUncompressedPages;
1899
1900 HIBLOG("hibernate_machine_init: state %d, image pages %d, sum was %x, image1Size %qx, conflictCount %d, nextFree %x\n",
1901 gIOHibernateState, pagesDone, sum, gIOHibernateCurrentHeader->image1Size,
1902 gIOHibernateCurrentHeader->conflictCount, gIOHibernateCurrentHeader->nextFree);
1903
1904 if (kIOHibernateStateWakingFromHibernate != gIOHibernateState)
1905 {
1906 HIBLOG("regular wake\n");
1907 return;
1908 }
1909
1910 HIBPRINT("diag %x %x %x %x\n",
1911 gIOHibernateCurrentHeader->diag[0], gIOHibernateCurrentHeader->diag[1],
1912 gIOHibernateCurrentHeader->diag[2], gIOHibernateCurrentHeader->diag[3]);
1913
1914 HIBPRINT("video %x %d %d %d\n",
1915 gIOHibernateGraphicsInfo->physicalAddress, gIOHibernateGraphicsInfo->depth,
1916 gIOHibernateGraphicsInfo->width, gIOHibernateGraphicsInfo->height);
1917
1918 if (vars->videoMapping && gIOHibernateGraphicsInfo->physicalAddress)
1919 {
1920 vars->videoMapSize = round_page(gIOHibernateGraphicsInfo->height
1921 * gIOHibernateGraphicsInfo->rowBytes);
1922 IOMapPages(kernel_map,
1923 vars->videoMapping, gIOHibernateGraphicsInfo->physicalAddress,
1924 vars->videoMapSize, kIOMapInhibitCache );
1925 }
1926
1927 uint8_t * src = (uint8_t *) vars->srcBuffer->getBytesNoCopy();;
1928 uint32_t decoOffset;
1929
1930 clock_get_uptime(&allTime);
1931
1932 HIBLOG("IOHibernatePollerOpen(), ml_get_interrupts_enabled %d\n", ml_get_interrupts_enabled());
1933 err = IOHibernatePollerOpen(vars->fileVars, kIOPolledAfterSleepState, 0);
1934 HIBLOG("IOHibernatePollerOpen(%x)\n", err);
1935
1936 if (gIOHibernateCurrentHeader->previewSize)
1937 progressZeroPosition = gIOHibernateCurrentHeader->previewSize
1938 + gIOHibernateCurrentHeader->fileExtentMapSize
1939 - sizeof(gIOHibernateCurrentHeader->fileExtentMap)
1940 + ptoa_64(gIOHibernateCurrentHeader->restore1PageCount);
1941
1942 IOPolledFileSeek(vars->fileVars, gIOHibernateCurrentHeader->image1Size);
1943
1944 if (vars->videoMapping)
1945 {
1946 lastBlob = ((vars->fileVars->position - progressZeroPosition) * kIOHibernateProgressCount)
1947 / (gIOHibernateCurrentHeader->imageSize - progressZeroPosition);
1948 ProgressUpdate(gIOHibernateGraphicsInfo, (uint8_t *) vars->videoMapping, 0, lastBlob);
1949 }
1950
1951 cryptvars = (kIOHibernateModeEncrypt & gIOHibernateMode) ? &gIOHibernateCryptWakeContext : 0;
1952 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1953 {
1954 cryptvars = &gIOHibernateCryptWakeContext;
1955 bcopy(&gIOHibernateCryptWakeVars->aes_iv[0],
1956 &cryptvars->aes_iv[0],
1957 sizeof(cryptvars->aes_iv));
1958 }
1959
1960 // kick off the read ahead
1961 vars->fileVars->io = false;
1962 vars->fileVars->bufferHalf = 0;
1963 vars->fileVars->bufferLimit = 0;
1964 vars->fileVars->lastRead = 0;
1965 vars->fileVars->bufferOffset = vars->fileVars->bufferLimit;
1966
1967 IOPolledFileRead(vars->fileVars, 0, 0, cryptvars);
1968 vars->fileVars->bufferOffset = vars->fileVars->bufferLimit;
1969 // --
1970
1971 HIBLOG("hibernate_machine_init reading\n");
1972
1973 uint32_t * header = (uint32_t *) src;
1974 sum = 0;
1975
1976 do
1977 {
1978 unsigned int count;
1979 unsigned int page;
1980 uint32_t tag;
1981 vm_offset_t ppnum, compressedSize;
1982
1983 IOPolledFileRead(vars->fileVars, src, 8, cryptvars);
1984
1985 ppnum = header[0];
1986 count = header[1];
1987
1988// HIBPRINT("(%x, %x)\n", ppnum, count);
1989
1990 if (!count)
1991 break;
1992
1993 for (page = 0; page < count; page++)
1994 {
1995 IOPolledFileRead(vars->fileVars, (uint8_t *) &tag, 4, cryptvars);
1996
1997 compressedSize = kIOHibernateTagLength & tag;
1998 if (!compressedSize)
1999 {
2000 ppnum++;
2001 pagesDone++;
2002 continue;
2003 }
2004
2005 IOPolledFileRead(vars->fileVars, src, (compressedSize + 3) & ~3, cryptvars);
2006
2007 if (compressedSize != page_size)
2008 {
2009 decoOffset = page_size;
2010 WKdm_decompress((WK_word*) src, (WK_word*) (src + decoOffset), PAGE_SIZE_IN_WORDS);
2011 }
2012 else
2013 decoOffset = 0;
2014
2015 sum += hibernate_sum((src + decoOffset), page_size);
2016
2017 err = IOMemoryDescriptorReadToPhysical(vars->srcBuffer, decoOffset, ptoa_64(ppnum), page_size);
2018 if (err)
2019 HIBLOG("IOMemoryDescriptorReadToPhysical [%d] %x\n", ppnum, err);
2020
2021 ppnum++;
2022 pagesDone++;
2023
2024 if (vars->videoMapping && (0 == (255 & pagesDone)))
2025 {
2026 blob = ((vars->fileVars->position - progressZeroPosition) * kIOHibernateProgressCount)
2027 / (gIOHibernateCurrentHeader->imageSize - progressZeroPosition);
2028 if (blob != lastBlob)
2029 {
2030 ProgressUpdate(gIOHibernateGraphicsInfo, (uint8_t *) vars->videoMapping, lastBlob, blob);
2031 lastBlob = blob;
2032 }
2033 }
2034
2035 if (0 == (8191 & pagesDone))
2036 {
2037 clock_get_uptime(&endTime);
2038 SUB_ABSOLUTETIME(&endTime, &allTime);
2039 absolutetime_to_nanoseconds(endTime, &nsec);
2040 progressStamp = nsec / 750000000ULL;
2041 if (progressStamp != lastProgressStamp)
2042 {
2043 lastProgressStamp = progressStamp;
2044 HIBPRINT("pages %d (%d%%)\n", pagesDone,
2045 (100 * pagesDone) / gIOHibernateCurrentHeader->pageCount);
2046 }
2047 }
2048 }
2049 }
2050 while (true);
2051
2052 gIOHibernateCurrentHeader->actualImage2Sum = sum;
2053
2054 if (vars->fileVars->io)
2055 (void) IOHibernatePollerIODone(vars->fileVars);
2056
2057 err = IOHibernatePollerClose(vars->fileVars, kIOPolledAfterSleepState);
2058
2059 if (vars->videoMapping)
2060 ProgressUpdate(gIOHibernateGraphicsInfo,
2061 (uint8_t *) vars->videoMapping, 0, kIOHibernateProgressCount);
2062
2063 clock_get_uptime(&endTime);
2064 SUB_ABSOLUTETIME(&endTime, &allTime);
2065 absolutetime_to_nanoseconds(endTime, &nsec);
2066
2067 HIBLOG("hibernate_machine_init pagesDone %d sum2 %x, time: %qd ms\n",
2068 pagesDone, sum, nsec / 1000000ULL);
2069}
2070
2071/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
mirror of XNU