]> git.saurik.com Git - apple/xnu.git/blame - iokit/Kernel/IOHibernateIO.cpp
projects / apple / xnu.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
xnu-1504.9.17.tar.gz
[apple/xnu.git] / iokit / Kernel / IOHibernateIO.cpp
CommitLineData
3a60a9f5 1/*
b0d623f7 2 * Copyright (c) 2004-2008 Apple Computer, Inc. All rights reserved.
3a60a9f5 3 *
2d21ac55 4 * @APPLE_OSREFERENCE_LICENSE_HEADER_START@
3a60a9f5 5 *
2d21ac55
A		 6 * This file contains Original Code and/or Modifications of Original Code
7 * as defined in and that are subject to the Apple Public Source License
8 * Version 2.0 (the 'License'). You may not use this file except in
9 * compliance with the License. The rights granted to you under the License
10 * may not be used to create, or enable the creation or redistribution of,
11 * unlawful or unlicensed copies of an Apple operating system, or to
12 * circumvent, violate, or enable the circumvention or violation of, any
13 * terms of an Apple operating system software license agreement.
8f6c56a5 14 *
2d21ac55
A		 15 * Please obtain a copy of the License at
16 * http://www.opensource.apple.com/apsl/ and read it before using this file.
17 *
18 * The Original Code and all software distributed under the License are
19 * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
8f6c56a5
A		 20 * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
21 * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
2d21ac55
A		 22 * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
23 * Please see the License for the specific language governing rights and
24 * limitations under the License.
8f6c56a5 25 *
2d21ac55 26 * @APPLE_OSREFERENCE_LICENSE_HEADER_END@
3a60a9f5
A		 27 */
28
29
30/*
31
32Sleep:
33
34- PMRootDomain calls IOHibernateSystemSleep() before system sleep
35(devices awake, normal execution context)
36- IOHibernateSystemSleep opens the hibernation file (or partition) at the bsd level,
37 grabs its extents and searches for a polling driver willing to work with that IOMedia.
38 The BSD code makes an ioctl to the storage driver to get the partition base offset to
39 the disk, and other ioctls to get the transfer constraints
40 If successful, the file is written to make sure its initially not bootable (in case of
41 later failure) and nvram set to point to the first block of the file. (Has to be done
42 here so blocking is possible in nvram support).
43 hibernate_setup() in osfmk is called to allocate page bitmaps for all dram, and
44 page out any pages it wants to (currently zero, but probably some percentage of memory).
45 Its assumed just allocating pages will cause the VM system to naturally select the best
46 pages for eviction. It also copies processor flags needed for the restore path and sets
47 a flag in the boot processor proc info.
48 gIOHibernateState = kIOHibernateStateHibernating.
49- Regular sleep progresses - some drivers may inspect the root domain property
50 kIOHibernateStateKey to modify behavior. The platform driver saves state to memory
51 as usual but leaves motherboard I/O on.
52- Eventually the platform calls ml_ppc_sleep() in the shutdown context on the last cpu,
53 at which point memory is ready to be saved. mapping_hibernate_flush() is called to get
54 all ppc RC bits out of the hash table and caches into the mapping structures.
55- hibernate_write_image() is called (still in shutdown context, no blocking or preemption).
56 hibernate_page_list_setall() is called to get a bitmap of dram pages that need to be saved.
57 All pages are assumed to be saved (as part of the wired image) unless explicitly subtracted
58 by hibernate_page_list_setall(), avoiding having to find arch dependent low level bits.
59 The image header and block list are written. The header includes the second file extent so
60 only the header block is needed to read the file, regardless of filesystem.
61 The kernel section "__HIB" is written uncompressed to the image. This section of code and data
62 (only) is used to decompress the image during wake/boot.
63 Some additional pages are removed from the bitmaps - the buffers used for hibernation.
64 The bitmaps are written to the image.
65 More areas are removed from the bitmaps (after they have been written to the image) - the
66 section "__HIB" pages and interrupt stack.
67 Each wired page is compressed and written and then each non-wired page. Compression and
68 disk writes are in parallel.
69 The image header is written to the start of the file and the polling driver closed.
70 The machine powers down (or sleeps).
71
72Boot/Wake:
73
74- BootX sees the boot-image nvram variable containing the device and block number of the image,
75 reads the header and if the signature is correct proceeds. The boot-image variable is cleared.
76- BootX reads the portion of the image used for wired pages, to memory. Its assumed this will fit
77 in the OF memory environment, and the image is decrypted. There is no decompression in BootX,
78 that is in the kernel's __HIB section.
79- BootX copies the "__HIB" section to its correct position in memory, quiesces and calls its entry
80 hibernate_kernel_entrypoint(), passing the location of the image in memory. Translation is off,
81 only code & data in that section is safe to call since all the other wired pages are still
82 compressed in the image.
83- hibernate_kernel_entrypoint() removes pages occupied by the raw image from the page bitmaps.
84 It uses the bitmaps to work out which pages can be uncompressed from the image to their final
85 location directly, and copies those that can't to interim free pages. When the image has been
86 completed, the copies are uncompressed, overwriting the wired image pages.
87 hibernate_restore_phys_page() (in osfmk since its arch dependent, but part of the "__HIB" section)
88 is used to get pages into place for 64bit.
89- the reset vector is called (at least on ppc), the kernel proceeds on a normal wake, with some
90 changes conditional on the per proc flag - before VM is turned on the boot cpu, all mappings
91 are removed from the software strutures, and the hash table is reinitialized.
92- After the platform CPU init code is called, hibernate_machine_init() is called to restore the rest
93 of memory, using the polled mode driver, before other threads can run or any devices are turned on.
94 This reduces the memory usage for BootX and allows decompression in parallel with disk reads,
95 for the remaining non wired pages.
96- The polling driver is closed down and regular wake proceeds. When the kernel calls iokit to wake
97 (normal execution context) hibernate_teardown() in osmfk is called to release any memory, the file
98 is closed via bsd.
99
100Polled Mode I/O:
101
102IOHibernateSystemSleep() finds a polled mode interface to the ATA controller via a property in the
103registry, specifying an object of calls IOPolledInterface.
104
105Before the system goes to sleep it searches from the IOMedia object (could be a filesystem or
106partition) that the image is going to live, looking for polled interface properties. If it finds
107one the IOMedia object is passed to a "probe" call for the interface to accept or reject. All the
108interfaces found are kept in an ordered list.
109
110There is an Open/Close pair of calls made to each of the interfaces at various stages since there are
111few different contexts things happen in:
112
113- there is an Open/Close (Preflight) made before any part of the system has slept (I/O is all
114up and running) and after wake - this is safe to allocate memory and do anything. The device
115ignores sleep requests from that point since its a waste of time if it goes to sleep and
116immediately wakes back up for the image write.
117
118- there is an Open/Close (BeforeSleep) pair made around the image write operations that happen
119immediately before sleep. These can't block or allocate memory - the I/O system is asleep apart
120from the low level bits (motherboard I/O etc). There is only one thread running. The close can be
121used to flush and set the disk to sleep.
122
123- there is an Open/Close (AfterSleep) pair made around the image read operations that happen
124immediately after sleep. These can't block or allocate memory. This is happening after the platform
125expert has woken the low level bits of the system, but most of the I/O system has not. There is only
126one thread running.
127
128For the actual I/O, all the ops are with respect to a single IOMemoryDescriptor that was passed
129(prepared) to the Preflight Open() call. There is a read/write op, buffer offset to the IOMD for
130the data, an offset to the disk and length (block aligned 64 bit numbers), and completion callback.
131Each I/O is async but only one is ever outstanding. The polled interface has a checkForWork call
132that is called for the hardware to check for events, and complete the I/O via the callback.
133The hibernate path uses the same transfer constraints the regular cluster I/O path in BSD uses
134to restrict I/O ops.
135*/
136
137#include <sys/systm.h>
138
139#include <IOKit/IOWorkLoop.h>
140#include <IOKit/IOCommandGate.h>
141#include <IOKit/IOTimerEventSource.h>
142#include <IOKit/IOPlatformExpert.h>
143#include <IOKit/IOKitDebug.h>
144#include <IOKit/IOTimeStamp.h>
145#include <IOKit/pwr_mgt/RootDomain.h>
146#include <IOKit/pwr_mgt/IOPMPrivate.h>
147#include <IOKit/IOMessage.h>
148#include <IOKit/IODeviceTreeSupport.h>
149#include <IOKit/IOBSD.h>
150#include "RootDomainUserClient.h"
151#include <IOKit/pwr_mgt/IOPowerConnection.h>
152#include "IOPMPowerStateQueue.h"
153#include <IOKit/IOBufferMemoryDescriptor.h>
154#include <crypto/aes.h>
155
156#include <sys/uio.h>
157#include <sys/conf.h>
158#include <sys/stat.h>
159#include <sys/fcntl.h> // (FWRITE, ...)
3a60a9f5 160#include <sys/sysctl.h>
3a60a9f5
A		 161
162#include <IOKit/IOHibernatePrivate.h>
163#include <IOKit/IOPolledInterface.h>
164#include <IOKit/IONVRAM.h>
165#include "IOHibernateInternal.h"
166#include "WKdm.h"
167#include "IOKitKernelInternal.h"
168
169/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
170
0b4c1975 171extern unsigned int save_kdebug_enable;
3a60a9f5
A		 172extern uint32_t gIOHibernateState;
173uint32_t gIOHibernateMode;
174static char gIOHibernateBootSignature[256+1];
175static char gIOHibernateFilename[MAXPATHLEN+1];
176static uint32_t gIOHibernateFreeRatio = 0; // free page target (percent)
177uint32_t gIOHibernateFreeTime = 0*1000; // max time to spend freeing pages (ms)
178
179static IODTNVRAM * gIOOptionsEntry;
180static IORegistryEntry * gIOChosenEntry;
b0d623f7 181#if defined(__i386__) || defined(__x86_64__)
0c530ab8
A		 182static const OSSymbol * gIOCreateEFIDevicePathSymbol;
183#endif
3a60a9f5
A		 184
185static IOPolledFileIOVars gFileVars;
186static IOHibernateVars gIOHibernateVars;
187static struct kern_direct_file_io_ref_t * gIOHibernateFileRef;
188static hibernate_cryptvars_t gIOHibernateCryptWakeContext;
189
190/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
191
192enum { kXPRamAudioVolume = 8 };
193enum { kDefaultIOSize = 128 * 1024 };
194enum { kVideoMapSize = 32 * 1024 * 1024 };
195
196#ifndef kIOMediaPreferredBlockSizeKey
197#define kIOMediaPreferredBlockSizeKey "Preferred Block Size"
198#endif
199
200#ifndef kIOBootPathKey
201#define kIOBootPathKey "bootpath"
202#endif
203#ifndef kIOSelectedBootDeviceKey
204#define kIOSelectedBootDeviceKey "boot-device"
205#endif
206
207
208enum { kIOHibernateMinPollersNeeded = 2 };
209
210/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
211
212// copy from phys addr to MD
213
214static IOReturn
215IOMemoryDescriptorWriteFromPhysical(IOMemoryDescriptor * md,
216 IOByteCount offset, addr64_t bytes, IOByteCount length)
217{
218 addr64_t srcAddr = bytes;
219 IOByteCount remaining;
220
221 remaining = length = min(length, md->getLength() - offset);
222 while (remaining) { // (process another target segment?)
223 addr64_t dstAddr64;
224 IOByteCount dstLen;
225
b0d623f7 226 dstAddr64 = md->getPhysicalSegment(offset, &dstLen, kIOMemoryMapperNone);
3a60a9f5
A		 227 if (!dstAddr64)
228 break;
229
230 // Clip segment length to remaining
231 if (dstLen > remaining)
232 dstLen = remaining;
233
234#if 1
235 bcopy_phys(srcAddr, dstAddr64, dstLen);
236#else
237 copypv(srcAddr, dstAddr64, dstLen,
238 cppvPsnk | cppvFsnk | cppvNoRefSrc | cppvNoModSnk | cppvKmap);
239#endif
240 srcAddr += dstLen;
241 offset += dstLen;
242 remaining -= dstLen;
243 }
244
245 assert(!remaining);
246
247 return remaining ? kIOReturnUnderrun : kIOReturnSuccess;
248}
249
250// copy from MD to phys addr
251
252static IOReturn
253IOMemoryDescriptorReadToPhysical(IOMemoryDescriptor * md,
254 IOByteCount offset, addr64_t bytes, IOByteCount length)
255{
256 addr64_t dstAddr = bytes;
257 IOByteCount remaining;
258
259 remaining = length = min(length, md->getLength() - offset);
260 while (remaining) { // (process another target segment?)
261 addr64_t srcAddr64;
262 IOByteCount dstLen;
263
b0d623f7 264 srcAddr64 = md->getPhysicalSegment(offset, &dstLen, kIOMemoryMapperNone);
3a60a9f5
A		 265 if (!srcAddr64)
266 break;
267
268 // Clip segment length to remaining
269 if (dstLen > remaining)
270 dstLen = remaining;
271
272#if 1
273 bcopy_phys(srcAddr64, dstAddr, dstLen);
274#else
275 copypv(srcAddr, dstAddr64, dstLen,
276 cppvPsnk | cppvFsnk | cppvNoRefSrc | cppvNoModSnk | cppvKmap);
277#endif
278 dstAddr += dstLen;
279 offset += dstLen;
280 remaining -= dstLen;
281 }
282
283 assert(!remaining);
284
285 return remaining ? kIOReturnUnderrun : kIOReturnSuccess;
286}
287
288/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
289
290void
291hibernate_set_page_state(hibernate_page_list_t * page_list, hibernate_page_list_t * page_list_wired,
292 vm_offset_t ppnum, vm_offset_t count, uint32_t kind)
293{
294 count += ppnum;
295 switch (kind)
296 {
297 case kIOHibernatePageStateUnwiredSave:
298 // unwired save
299 for (; ppnum < count; ppnum++)
300 {
301 hibernate_page_bitset(page_list, FALSE, ppnum);
302 hibernate_page_bitset(page_list_wired, TRUE, ppnum);
303 }
304 break;
305 case kIOHibernatePageStateWiredSave:
306 // wired save
307 for (; ppnum < count; ppnum++)
308 {
309 hibernate_page_bitset(page_list, FALSE, ppnum);
310 hibernate_page_bitset(page_list_wired, FALSE, ppnum);
311 }
312 break;
313 case kIOHibernatePageStateFree:
314 // free page
315 for (; ppnum < count; ppnum++)
316 {
317 hibernate_page_bitset(page_list, TRUE, ppnum);
318 hibernate_page_bitset(page_list_wired, TRUE, ppnum);
319 }
320 break;
321 default:
322 panic("hibernate_set_page_state");
323 }
324}
325
326static vm_offset_t
0c530ab8 327hibernate_page_list_iterate(hibernate_page_list_t * list, vm_offset_t * pPage)
3a60a9f5 328{
0c530ab8
A		 329 uint32_t page = *pPage;
330 uint32_t count;
331 hibernate_bitmap_t * bitmap;
21362eb3 332
0c530ab8
A		 333 while ((bitmap = hibernate_page_bitmap_pin(list, &page)))
334 {
335 count = hibernate_page_bitmap_count(bitmap, TRUE, page);
336 if (!count)
337 break;
338 page += count;
339 if (page <= bitmap->last_page)
340 break;
341 }
6601e61a 342
0c530ab8
A		 343 *pPage = page;
344 if (bitmap)
345 count = hibernate_page_bitmap_count(bitmap, FALSE, page);
346 else
347 count = 0;
3a60a9f5
A		 348
349 return (count);
350}
351
352/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
353
354static IOReturn
355IOHibernatePollerProbe(IOPolledFileIOVars * vars, IOService * target)
356{
357 IOReturn err = kIOReturnError;
358 int32_t idx;
359 IOPolledInterface * poller;
360
361 for (idx = vars->pollers->getCount() - 1; idx >= 0; idx--)
362 {
363 poller = (IOPolledInterface *) vars->pollers->getObject(idx);
364 err = poller->probe(target);
365 if (err)
366 {
367 HIBLOG("IOPolledInterface::probe[%d] 0x%x\n", idx, err);
368 break;
369 }
370 }
371
372 return (err);
373}
374
375static IOReturn
376IOHibernatePollerOpen(IOPolledFileIOVars * vars, uint32_t state, IOMemoryDescriptor * md)
377{
378 IOReturn err = kIOReturnError;
379 int32_t idx;
380 IOPolledInterface * poller;
381
382 for (idx = vars->pollers->getCount() - 1; idx >= 0; idx--)
383 {
384 poller = (IOPolledInterface *) vars->pollers->getObject(idx);
385 err = poller->open(state, md);
386 if (err)
387 {
388 HIBLOG("IOPolledInterface::open[%d] 0x%x\n", idx, err);
389 break;
390 }
391 }
392
393 return (err);
394}
395
396static IOReturn
397IOHibernatePollerClose(IOPolledFileIOVars * vars, uint32_t state)
398{
399 IOReturn err = kIOReturnError;
400 int32_t idx;
401 IOPolledInterface * poller;
402
403 for (idx = 0;
404 (poller = (IOPolledInterface *) vars->pollers->getObject(idx));
405 idx++)
406 {
407 err = poller->close(state);
408 if (err)
409 HIBLOG("IOPolledInterface::close[%d] 0x%x\n", idx, err);
410 }
411
412 return (err);
413}
414
415static void
416IOHibernatePollerIOComplete(void * target,
417 void * parameter,
418 IOReturn status,
419 UInt64 actualByteCount)
420{
421 IOPolledFileIOVars * vars = (IOPolledFileIOVars *) parameter;
422
423 vars->ioStatus = status;
424}
425
426static IOReturn
427IOHibernatePollerIO(IOPolledFileIOVars * vars,
428 uint32_t operation, uint32_t bufferOffset,
429 uint64_t deviceOffset, uint64_t length)
430{
431
432 IOReturn err = kIOReturnError;
433 IOPolledInterface * poller;
434 IOPolledCompletion completion;
435
436 completion.target = 0;
437 completion.action = &IOHibernatePollerIOComplete;
438 completion.parameter = vars;
439
440 vars->ioStatus = -1;
441
442 poller = (IOPolledInterface *) vars->pollers->getObject(0);
443 err = poller->startIO(operation, bufferOffset, deviceOffset + vars->block0, length, completion);
444 if (err)
445 HIBLOG("IOPolledInterface::startIO[%d] 0x%x\n", 0, err);
446
447 return (err);
448}
449
450static IOReturn
2d21ac55 451IOHibernatePollerIODone(IOPolledFileIOVars * vars, bool abortable)
3a60a9f5 452{
2d21ac55
A		 453 IOReturn err = kIOReturnSuccess;
454 int32_t idx = 0;
3a60a9f5
A		 455 IOPolledInterface * poller;
456
457 while (-1 == vars->ioStatus)
458 {
2d21ac55
A		 459 for (idx = 0;
460 (poller = (IOPolledInterface *) vars->pollers->getObject(idx));
3a60a9f5
A		 461 idx++)
462 {
2d21ac55
A		 463 IOReturn newErr;
464 newErr = poller->checkForWork();
465 if ((newErr == kIOReturnAborted) && !abortable)
466 newErr = kIOReturnSuccess;
467 if (kIOReturnSuccess == err)
468 err = newErr;
3a60a9f5
A		 469 }
470 }
471
0b4c1975
A		 472 if ((kIOReturnSuccess == err) && abortable && hibernate_should_abort())
473 {
474 err = kIOReturnAborted;
475 HIBLOG("IOPolledInterface::checkForWork sw abort\n");
476 }
477
2d21ac55
A		 478 if (err)
479 {
480 HIBLOG("IOPolledInterface::checkForWork[%d] 0x%x\n", idx, err);
481 }
482 else
483 {
484 err = vars->ioStatus;
485 if (kIOReturnSuccess != err)
486 HIBLOG("IOPolledInterface::ioStatus 0x%x\n", err);
487 }
3a60a9f5 488
2d21ac55 489 return (err);
3a60a9f5
A		 490}
491
492IOReturn
493IOPolledInterface::checkAllForWork(void)
494{
495 IOReturn err = kIOReturnNotReady;
496 int32_t idx;
497 IOPolledInterface * poller;
498
499 IOHibernateVars * vars = &gIOHibernateVars;
500
501 if (!vars->fileVars || !vars->fileVars->pollers)
502 return (err);
503
504 for (idx = 0;
505 (poller = (IOPolledInterface *) vars->fileVars->pollers->getObject(idx));
506 idx++)
507 {
508 err = poller->checkForWork();
509 if (err)
510 HIBLOG("IOPolledInterface::checkAllForWork[%d] 0x%x\n", idx, err);
511 }
512
513 return (err);
514}
515
516/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
517
518struct _OpenFileContext
519{
520 OSData * extents;
521 uint64_t size;
522};
523
524static void
525file_extent_callback(void * ref, uint64_t start, uint64_t length)
526{
527 _OpenFileContext * ctx = (_OpenFileContext *) ref;
528 IOPolledFileExtent extent;
529
530 extent.start = start;
531 extent.length = length;
532
533 ctx->extents->appendBytes(&extent, sizeof(extent));
534 ctx->size += length;
535}
536
537IOReturn
538IOPolledFileOpen( const char * filename, IOBufferMemoryDescriptor * ioBuffer,
539 IOPolledFileIOVars ** fileVars, OSData ** fileExtents,
540 OSData ** imagePath)
541{
542 IOReturn err = kIOReturnError;
543 IOPolledFileIOVars * vars;
544 _OpenFileContext ctx;
545 OSData * extentsData;
546 OSNumber * num;
547 IORegistryEntry * part = 0;
548 OSDictionary * matching;
549 OSIterator * iter;
550 dev_t hibernate_image_dev;
551 uint64_t maxiobytes;
552
553 vars = &gFileVars;
554 do
555 {
556 HIBLOG("sizeof(IOHibernateImageHeader) == %ld\n", sizeof(IOHibernateImageHeader));
557 if (sizeof(IOHibernateImageHeader) != 512)
558 continue;
559
560 vars->io = false;
561 vars->buffer = (uint8_t *) ioBuffer->getBytesNoCopy();
562 vars->bufferHalf = 0;
563 vars->bufferOffset = 0;
564 vars->bufferSize = ioBuffer->getLength() >> 1;
565
566 extentsData = OSData::withCapacity(32);
567
568 ctx.extents = extentsData;
569 ctx.size = 0;
570 vars->fileRef = kern_open_file_for_direct_io(filename,
571 &file_extent_callback, &ctx,
572 &hibernate_image_dev,
573 &vars->block0,
0b4c1975
A		 574 &maxiobytes,
575 &vars->solid_state);
3a60a9f5
A		 576 if (!vars->fileRef)
577 {
578 err = kIOReturnNoSpace;
579 break;
580 }
b0d623f7 581 gIOHibernateFileRef = vars->fileRef;
0b4c1975
A		 582
583 if (kIOHibernateModeSSDInvert & gIOHibernateMode)
584 vars->solid_state = vars->solid_state ? false : true;
585
586 HIBLOG("Opened file %s, size %qd, partition base 0x%qx, maxio %qx ssd %d\n", filename, ctx.size,
587 vars->block0, maxiobytes, vars->solid_state);
3a60a9f5
A		 588 if (ctx.size < 1*1024*1024) // check against image size estimate!
589 {
590 err = kIOReturnNoSpace;
591 break;
592 }
593
594 if (maxiobytes < vars->bufferSize)
595 vars->bufferSize = maxiobytes;
596
597 vars->extentMap = (IOPolledFileExtent *) extentsData->getBytesNoCopy();
598
599 matching = IOService::serviceMatching("IOMedia");
600 num = OSNumber::withNumber(major(hibernate_image_dev), 32);
601 matching->setObject(kIOBSDMajorKey, num);
602 num->release();
603 num = OSNumber::withNumber(minor(hibernate_image_dev), 32);
604 matching->setObject(kIOBSDMinorKey, num);
605 num->release();
606 iter = IOService::getMatchingServices(matching);
607 matching->release();
608 if (iter)
609 {
610 part = (IORegistryEntry *) iter->getNextObject();
611 part->retain();
612 iter->release();
613 }
2d21ac55
A		 614 if (!part)
615 break;
616
3a60a9f5
A		 617 int minor, major;
618 IORegistryEntry * next;
619 IORegistryEntry * child;
620 OSData * data;
621
622 num = (OSNumber *) part->getProperty(kIOBSDMajorKey);
623 if (!num)
624 break;
625 major = num->unsigned32BitValue();
626 num = (OSNumber *) part->getProperty(kIOBSDMinorKey);
627 if (!num)
628 break;
629 minor = num->unsigned32BitValue();
630
631 hibernate_image_dev = makedev(major, minor);
632
633 vars->pollers = OSArray::withCapacity(4);
634 if (!vars->pollers)
635 break;
636
637 vars->blockSize = 512;
638 next = part;
639 do
640 {
641 IOPolledInterface * poller;
c0fea474
A		 642 OSObject * obj;
643
644 obj = next->getProperty(kIOPolledInterfaceSupportKey);
645 if (kOSBooleanFalse == obj)
646 {
647 vars->pollers->flushCollection();
648 break;
649 }
650 else if ((poller = OSDynamicCast(IOPolledInterface, obj)))
3a60a9f5
A		 651 vars->pollers->setObject(poller);
652 if ((num = OSDynamicCast(OSNumber, next->getProperty(kIOMediaPreferredBlockSizeKey))))
653 vars->blockSize = num->unsigned32BitValue();
654 child = next;
655 }
656 while ((next = child->getParentEntry(gIOServicePlane))
657 && child->isParent(next, gIOServicePlane, true));
658
659 HIBLOG("hibernate image major %d, minor %d, blocksize %ld, pollers %d\n",
b0d623f7 660 major, minor, (long)vars->blockSize, vars->pollers->getCount());
3a60a9f5
A		 661 if (vars->pollers->getCount() < kIOHibernateMinPollersNeeded)
662 continue;
663
664 err = IOHibernatePollerProbe(vars, (IOService *) part);
665 if (kIOReturnSuccess != err)
666 break;
667
668 err = IOHibernatePollerOpen(vars, kIOPolledPreflightState, ioBuffer);
669 if (kIOReturnSuccess != err)
670 break;
671
672 *fileVars = vars;
673 *fileExtents = extentsData;
674
675 // make imagePath
3a60a9f5 676
0c530ab8 677 if ((extentsData->getLength() >= sizeof(IOPolledFileExtent)))
3a60a9f5 678 {
0c530ab8
A		 679 char str2[24];
680
b0d623f7 681#if defined(__i386__) || defined(__x86_64__)
0c530ab8
A		 682 if (!gIOCreateEFIDevicePathSymbol)
683 gIOCreateEFIDevicePathSymbol = OSSymbol::withCString("CreateEFIDevicePath");
684
2d21ac55 685 snprintf(str2, sizeof(str2), "%qx", vars->extentMap[0].start);
0c530ab8
A		 686
687 err = IOService::getPlatform()->callPlatformFunction(
688 gIOCreateEFIDevicePathSymbol, false,
689 (void *) part, (void *) str2, (void *) true,
690 (void *) &data);
691#else
692 char str1[256];
693 int len = sizeof(str1);
694
695 if (!part->getPath(str1, &len, gIODTPlane))
696 err = kIOReturnNotFound;
697 else
698 {
2d21ac55 699 snprintf(str2, sizeof(str2), ",%qx", vars->extentMap[0].start);
0c530ab8
A		 700 // (strip the plane name)
701 char * tail = strchr(str1, ':');
702 if (!tail)
703 tail = str1 - 1;
704 data = OSData::withBytes(tail + 1, strlen(tail + 1));
705 data->appendBytes(str2, strlen(str2));
706 }
707#endif
708 if (kIOReturnSuccess == err)
3a60a9f5 709 *imagePath = data;
0c530ab8
A		 710 else
711 HIBLOG("error 0x%x getting path\n", err);
3a60a9f5
A		 712 }
713 }
714 while (false);
715
716 if (kIOReturnSuccess != err)
717 {
718 HIBLOG("error 0x%x opening hibernation file\n", err);
719 if (vars->fileRef)
b0d623f7 720 {
3a60a9f5 721 kern_close_file_for_direct_io(vars->fileRef);
b0d623f7
A		 722 gIOHibernateFileRef = vars->fileRef = NULL;
723 }
3a60a9f5
A		 724 }
725
726 if (part)
727 part->release();
728
729 return (err);
730}
731
732IOReturn
733IOPolledFileClose( IOPolledFileIOVars * vars )
734{
735 if (vars->pollers)
736 {
737 IOHibernatePollerClose(vars, kIOPolledPostflightState);
738 vars->pollers->release();
739 }
740
3a60a9f5
A		 741 bzero(vars, sizeof(IOPolledFileIOVars));
742
743 return (kIOReturnSuccess);
744}
745
746static IOReturn
747IOPolledFileSeek(IOPolledFileIOVars * vars, uint64_t position)
748{
749 IOPolledFileExtent * extentMap;
750
751 extentMap = vars->extentMap;
752
753 vars->position = position;
754
755 while (position >= extentMap->length)
756 {
757 position -= extentMap->length;
758 extentMap++;
759 }
760
761 vars->currentExtent = extentMap;
762 vars->extentRemaining = extentMap->length - position;
763 vars->extentPosition = vars->position - position;
764
765 if (vars->bufferSize <= vars->extentRemaining)
766 vars->bufferLimit = vars->bufferSize;
767 else
768 vars->bufferLimit = vars->extentRemaining;
769
770 return (kIOReturnSuccess);
771}
772
773static IOReturn
774IOPolledFileWrite(IOPolledFileIOVars * vars,
775 const uint8_t * bytes, IOByteCount size,
776 hibernate_cryptvars_t * cryptvars)
777{
778 IOReturn err = kIOReturnSuccess;
779 IOByteCount copy;
780 bool flush = false;
781
782 do
783 {
784 if (!bytes && !size)
785 {
786 // seek to end of block & flush
787 size = vars->position & (vars->blockSize - 1);
788 if (size)
789 size = vars->blockSize - size;
790 flush = true;
791 // use some garbage for the fill
792 bytes = vars->buffer + vars->bufferOffset;
793 }
794
795 copy = vars->bufferLimit - vars->bufferOffset;
796 if (copy > size)
797 copy = size;
798 else
799 flush = true;
800
801 if (bytes)
802 {
803 bcopy(bytes, vars->buffer + vars->bufferHalf + vars->bufferOffset, copy);
804 bytes += copy;
805 }
806 else
807 bzero(vars->buffer + vars->bufferHalf + vars->bufferOffset, copy);
808
809 size -= copy;
810 vars->bufferOffset += copy;
811 vars->position += copy;
812
813 if (flush && vars->bufferOffset)
814 {
815 uint64_t offset = (vars->position - vars->bufferOffset
816 - vars->extentPosition + vars->currentExtent->start);
817 uint32_t length = (vars->bufferOffset);
818
2d21ac55 819#if CRYPTO
0b4c1975
A		 820 if (cryptvars && vars->encryptStart
821 && (vars->position > vars->encryptStart)
822 && ((vars->position - length) < vars->encryptEnd))
3a60a9f5
A		 823 {
824 uint32_t encryptLen, encryptStart;
825 encryptLen = vars->position - vars->encryptStart;
826 if (encryptLen > length)
827 encryptLen = length;
828 encryptStart = length - encryptLen;
0b4c1975
A		 829 if (vars->position > vars->encryptEnd)
830 encryptLen -= (vars->position - vars->encryptEnd);
831
3a60a9f5
A		 832 // encrypt the buffer
833 aes_encrypt_cbc(vars->buffer + vars->bufferHalf + encryptStart,
834 &cryptvars->aes_iv[0],
835 encryptLen / AES_BLOCK_SIZE,
836 vars->buffer + vars->bufferHalf + encryptStart,
837 &cryptvars->ctx.encrypt);
838 // save initial vector for following encrypts
839 bcopy(vars->buffer + vars->bufferHalf + encryptStart + encryptLen - AES_BLOCK_SIZE,
840 &cryptvars->aes_iv[0],
841 AES_BLOCK_SIZE);
842 }
2d21ac55 843#endif /* CRYPTO */
3a60a9f5
A		 844
845 if (vars->io)
846 {
2d21ac55 847 err = IOHibernatePollerIODone(vars, true);
3a60a9f5
A		 848 if (kIOReturnSuccess != err)
849 break;
850 }
851
852if (vars->position & (vars->blockSize - 1)) HIBLOG("misaligned file pos %qx\n", vars->position);
853//if (length != vars->bufferSize) HIBLOG("short write of %qx ends@ %qx\n", length, offset + length);
854
855 err = IOHibernatePollerIO(vars, kIOPolledWrite, vars->bufferHalf, offset, length);
856 if (kIOReturnSuccess != err)
857 break;
858 vars->io = true;
859
860 vars->extentRemaining -= vars->bufferOffset;
861 if (!vars->extentRemaining)
862 {
863 vars->currentExtent++;
864 vars->extentRemaining = vars->currentExtent->length;
865 vars->extentPosition = vars->position;
866 if (!vars->extentRemaining)
867 {
868 err = kIOReturnOverrun;
869 break;
870 }
871 }
872
873 vars->bufferHalf = vars->bufferHalf ? 0 : vars->bufferSize;
874 vars->bufferOffset = 0;
875 if (vars->bufferSize <= vars->extentRemaining)
876 vars->bufferLimit = vars->bufferSize;
877 else
878 vars->bufferLimit = vars->extentRemaining;
879
880 flush = false;
881 }
882 }
883 while (size);
884
885 return (err);
886}
887
888static IOReturn
889IOPolledFileRead(IOPolledFileIOVars * vars,
890 uint8_t * bytes, IOByteCount size,
891 hibernate_cryptvars_t * cryptvars)
892{
893 IOReturn err = kIOReturnSuccess;
894 IOByteCount copy;
895
896// bytesWritten += size;
897
898 do
899 {
900 copy = vars->bufferLimit - vars->bufferOffset;
901 if (copy > size)
902 copy = size;
903
904 if (bytes)
905 {
906 bcopy(vars->buffer + vars->bufferHalf + vars->bufferOffset, bytes, copy);
907 bytes += copy;
908 }
909 size -= copy;
910 vars->bufferOffset += copy;
911// vars->position += copy;
912
913 if (vars->bufferOffset == vars->bufferLimit)
914 {
915 if (vars->io)
916 {
2d21ac55 917 err = IOHibernatePollerIODone(vars, false);
3a60a9f5
A		 918 if (kIOReturnSuccess != err)
919 break;
920 }
921 else
922 cryptvars = 0;
923
924if (vars->position & (vars->blockSize - 1)) HIBLOG("misaligned file pos %qx\n", vars->position);
925
926 vars->position += vars->lastRead;
927 vars->extentRemaining -= vars->lastRead;
928 vars->bufferLimit = vars->lastRead;
929
930 if (!vars->extentRemaining)
931 {
932 vars->currentExtent++;
933 vars->extentRemaining = vars->currentExtent->length;
934 vars->extentPosition = vars->position;
935 if (!vars->extentRemaining)
936 {
937 err = kIOReturnOverrun;
938 break;
939 }
940 }
941
36401178
A		 942 uint64_t length;
943 uint64_t lastReadLength = vars->lastRead;
3a60a9f5
A		 944 uint64_t offset = (vars->position
945 - vars->extentPosition + vars->currentExtent->start);
36401178
A		 946 if (vars->extentRemaining <= vars->bufferSize)
947 length = vars->extentRemaining;
948 else
949 length = vars->bufferSize;
950 vars->lastRead = length;
3a60a9f5
A		 951
952//if (length != vars->bufferSize) HIBLOG("short read of %qx ends@ %qx\n", length, offset + length);
953
954 err = IOHibernatePollerIO(vars, kIOPolledRead, vars->bufferHalf, offset, length);
955 if (kIOReturnSuccess != err)
956 break;
957 vars->io = true;
958
959 vars->bufferHalf = vars->bufferHalf ? 0 : vars->bufferSize;
960 vars->bufferOffset = 0;
961
2d21ac55 962#if CRYPTO
3a60a9f5
A		 963 if (cryptvars)
964 {
965 uint8_t thisVector[AES_BLOCK_SIZE];
966 // save initial vector for following decrypts
967 bcopy(&cryptvars->aes_iv[0], &thisVector[0], AES_BLOCK_SIZE);
36401178 968 bcopy(vars->buffer + vars->bufferHalf + lastReadLength - AES_BLOCK_SIZE,
3a60a9f5
A		 969 &cryptvars->aes_iv[0], AES_BLOCK_SIZE);
970 // decrypt the buffer
971 aes_decrypt_cbc(vars->buffer + vars->bufferHalf,
972 &thisVector[0],
36401178 973 lastReadLength / AES_BLOCK_SIZE,
3a60a9f5
A		 974 vars->buffer + vars->bufferHalf,
975 &cryptvars->ctx.decrypt);
976 }
b0d623f7 977#endif /* CRYPTO */
3a60a9f5
A		 978 }
979 }
980 while (size);
981
982 return (err);
983}
984
985/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
986
987IOReturn
988IOHibernateSystemSleep(void)
989{
990 IOReturn err;
991 OSData * data;
992 OSObject * obj;
993 OSString * str;
0b4c1975 994 bool dsSSD;
3a60a9f5
A		 995
996 IOHibernateVars * vars = &gIOHibernateVars;
997
998 if (vars->fileVars && vars->fileVars->fileRef)
999 // already on the way down
1000 return (kIOReturnSuccess);
1001
1002 gIOHibernateState = kIOHibernateStateInactive;
1003
b0d623f7
A		 1004 gIOHibernateDebugFlags = 0;
1005 if (kIOLogHibernate & gIOKitDebug)
1006 gIOHibernateDebugFlags |= kIOHibernateDebugRestoreLogs;
1007
0b4c1975
A		 1008 if (IOService::getPMRootDomain()->getHibernateSettings(
1009 &gIOHibernateMode, &gIOHibernateFreeRatio, &gIOHibernateFreeTime))
3a60a9f5
A		 1010 if (kIOHibernateModeSleep & gIOHibernateMode)
1011 // default to discard clean for safe sleep
1012 gIOHibernateMode ^= (kIOHibernateModeDiscardCleanInactive
1013 | kIOHibernateModeDiscardCleanActive);
1014
3a60a9f5
A		 1015 if ((obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateFileKey)))
1016 {
1017 if ((str = OSDynamicCast(OSString, obj)))
2d21ac55
A		 1018 strlcpy(gIOHibernateFilename, str->getCStringNoCopy(),
1019 sizeof(gIOHibernateFilename));
3a60a9f5
A		 1020 obj->release();
1021 }
1022
1023 if (!gIOHibernateMode || !gIOHibernateFilename[0])
1024 return (kIOReturnUnsupported);
1025
1026 HIBLOG("hibernate image path: %s\n", gIOHibernateFilename);
1027
2d21ac55 1028
3a60a9f5
A		 1029 do
1030 {
0c530ab8
A		 1031 vars->srcBuffer = IOBufferMemoryDescriptor::withOptions(kIODirectionOutIn,
1032 4 * page_size, page_size);
1033 vars->ioBuffer = IOBufferMemoryDescriptor::withOptions(kIODirectionOutIn,
1034 2 * kDefaultIOSize, page_size);
3a60a9f5
A		 1035
1036 if (!vars->srcBuffer || !vars->ioBuffer)
1037 {
1038 err = kIOReturnNoMemory;
1039 break;
1040 }
1041
1042 err = IOPolledFileOpen(gIOHibernateFilename, vars->ioBuffer,
1043 &vars->fileVars, &vars->fileExtents, &data);
1044 if (KERN_SUCCESS != err)
1045 {
1046 HIBLOG("IOPolledFileOpen(%x)\n", err);
1047 break;
1048 }
1049 if (vars->fileVars->fileRef)
1050 {
1051 // invalidate the image file
1052 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderInvalidSignature;
1053 int err = kern_write_file(vars->fileVars->fileRef, 0,
1054 (caddr_t) gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
1055 if (KERN_SUCCESS != err)
1056 HIBLOG("kern_write_file(%d)\n", err);
1057 }
1058
1059 bzero(gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
b0d623f7 1060 gIOHibernateCurrentHeader->debugFlags = gIOHibernateDebugFlags;
3a60a9f5 1061
0b4c1975
A		 1062 dsSSD = (vars->fileVars->solid_state
1063 && (kOSBooleanTrue == IOService::getPMRootDomain()->getProperty(kIOPMDeepSleepEnabledKey)));
1064
1065 if (dsSSD)
1066 {
1067 gIOHibernateCurrentHeader->options |=
1068 kIOHibernateOptionSSD
1069 | kIOHibernateOptionColor;
1070 }
1071 else
1072 {
1073 gIOHibernateCurrentHeader->options |= kIOHibernateOptionProgress;
1074 }
1075
3a60a9f5 1076 boolean_t encryptedswap;
0b4c1975
A		 1077 AbsoluteTime startTime, endTime;
1078 uint64_t nsec;
1079
1080 clock_get_uptime(&startTime);
3a60a9f5
A		 1081 err = hibernate_setup(gIOHibernateCurrentHeader,
1082 gIOHibernateFreeRatio, gIOHibernateFreeTime,
0b4c1975 1083 dsSSD,
3a60a9f5 1084 &vars->page_list, &vars->page_list_wired, &encryptedswap);
0b4c1975
A		 1085 clock_get_uptime(&endTime);
1086 SUB_ABSOLUTETIME(&endTime, &startTime);
1087 absolutetime_to_nanoseconds(endTime, &nsec);
1088 HIBLOG("hibernate_setup(%d) took %qd ms\n", err, nsec / 1000000ULL);
1089
3a60a9f5 1090 if (KERN_SUCCESS != err)
3a60a9f5 1091 break;
3a60a9f5
A		 1092
1093 if (encryptedswap)
1094 gIOHibernateMode ^= kIOHibernateModeEncrypt;
1095
0b4c1975
A		 1096 if (kIOHibernateOptionProgress & gIOHibernateCurrentHeader->options)
1097 {
1098 vars->videoAllocSize = kVideoMapSize;
1099 if (KERN_SUCCESS != kmem_alloc_pageable(kernel_map, &vars->videoMapping, vars->videoAllocSize))
1100 vars->videoMapping = 0;
1101 }
3a60a9f5
A		 1102
1103 // generate crypt keys
1104 for (uint32_t i = 0; i < sizeof(vars->wiredCryptKey); i++)
1105 vars->wiredCryptKey[i] = random();
1106 for (uint32_t i = 0; i < sizeof(vars->cryptKey); i++)
1107 vars->cryptKey[i] = random();
1108
1109 // set nvram
1110
1111 IORegistryEntry * regEntry;
1112 if (!gIOOptionsEntry)
1113 {
1114 regEntry = IORegistryEntry::fromPath("/options", gIODTPlane);
1115 gIOOptionsEntry = OSDynamicCast(IODTNVRAM, regEntry);
1116 if (regEntry && !gIOOptionsEntry)
1117 regEntry->release();
1118 }
1119 if (!gIOChosenEntry)
1120 gIOChosenEntry = IORegistryEntry::fromPath("/chosen", gIODTPlane);
1121
1122 if (gIOOptionsEntry)
1123 {
1124 const OSSymbol * sym;
3a60a9f5
A		 1125
1126 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKey);
1127 if (sym)
1128 {
1129 gIOOptionsEntry->setProperty(sym, data);
1130 sym->release();
1131 }
1132 data->release();
1133
b0d623f7 1134#if defined(__ppc__)
0c530ab8
A		 1135 size_t len;
1136 char valueString[16];
1137
3a60a9f5
A		 1138 vars->saveBootDevice = gIOOptionsEntry->copyProperty(kIOSelectedBootDeviceKey);
1139 if (gIOChosenEntry)
1140 {
1141 OSData * bootDevice = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOBootPathKey));
1142 if (bootDevice)
1143 {
1144 sym = OSSymbol::withCStringNoCopy(kIOSelectedBootDeviceKey);
1145 OSString * str2 = OSString::withCStringNoCopy((const char *) bootDevice->getBytesNoCopy());
1146 if (sym && str2)
1147 gIOOptionsEntry->setProperty(sym, str2);
1148 if (sym)
1149 sym->release();
1150 if (str2)
1151 str2->release();
1152 }
3a60a9f5
A		 1153 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMemorySignatureKey));
1154 if (data)
1155 {
1156 vars->haveFastBoot = true;
1157
b0d623f7 1158 len = snprintf(valueString, sizeof(valueString), "0x%lx", *((UInt32 *)data->getBytesNoCopy()));
3a60a9f5
A		 1159 data = OSData::withBytes(valueString, len + 1);
1160 sym = OSSymbol::withCStringNoCopy(kIOHibernateMemorySignatureEnvKey);
1161 if (sym && data)
1162 gIOOptionsEntry->setProperty(sym, data);
1163 if (sym)
1164 sym->release();
1165 if (data)
1166 data->release();
1167 }
1168 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMachineSignatureKey));
1169 if (data)
1170 gIOHibernateCurrentHeader->machineSignature = *((UInt32 *)data->getBytesNoCopy());
1171 }
0c530ab8 1172#endif /* __ppc__ */
b0d623f7 1173#if defined(__i386__) || defined(__x86_64__)
0c530ab8
A		 1174 struct AppleRTCHibernateVars
1175 {
1176 uint8_t signature[4];
1177 uint32_t revision;
1178 uint8_t booterSignature[20];
1179 uint8_t wiredCryptKey[16];
1180 };
1181 AppleRTCHibernateVars rtcVars;
1182
1183 rtcVars.signature[0] = 'A';
1184 rtcVars.signature[1] = 'A';
1185 rtcVars.signature[2] = 'P';
1186 rtcVars.signature[3] = 'L';
1187 rtcVars.revision = 1;
1188 bcopy(&vars->wiredCryptKey[0], &rtcVars.wiredCryptKey[0], sizeof(rtcVars.wiredCryptKey));
1189 if (gIOHibernateBootSignature[0])
1190 {
1191 char c;
1192 uint8_t value = 0;
1193 for (uint32_t i = 0;
1194 (c = gIOHibernateBootSignature[i]) && (i < (sizeof(rtcVars.booterSignature) << 1));
1195 i++)
1196 {
1197 if (c >= 'a')
1198 c -= 'a' - 10;
1199 else if (c >= 'A')
1200 c -= 'A' - 10;
1201 else if (c >= '0')
1202 c -= '0';
1203 else
1204 continue;
1205 value = (value << 4) | c;
1206 if (i & 1)
1207 rtcVars.booterSignature[i >> 1] = value;
1208 }
1209 }
1210 data = OSData::withBytes(&rtcVars, sizeof(rtcVars));
1211 if (data)
2d21ac55
A		 1212 {
1213 IOService::getPMRootDomain()->setProperty(kIOHibernateRTCVariablesKey, data);
1214
1215 if( gIOOptionsEntry )
1216 {
1217 if( gIOHibernateMode & kIOHibernateModeSwitch )
1218 {
1219 const OSSymbol *sym;
1220 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootSwitchVarsKey);
1221 if( sym )
1222 {
1223 gIOOptionsEntry->setProperty(sym, data); /* intentional insecure backup of rtc boot vars */
1224 sym->release();
1225 }
1226 }
1227 }
1228
1229 data->release();
0c530ab8
A		 1230 }
1231 if (gIOChosenEntry)
1232 {
1233 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMachineSignatureKey));
1234 if (data)
1235 gIOHibernateCurrentHeader->machineSignature = *((UInt32 *)data->getBytesNoCopy());
1236 }
b0d623f7 1237#else /* !i386 && !x86_64 */
3a60a9f5
A		 1238 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1239 {
1240 data = OSData::withBytes(&vars->wiredCryptKey[0], sizeof(vars->wiredCryptKey));
1241 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKeyKey);
1242 if (sym && data)
1243 gIOOptionsEntry->setProperty(sym, data);
1244 if (sym)
1245 sym->release();
1246 if (data)
1247 data->release();
2d21ac55 1248 if (false && gIOHibernateBootSignature[0])
3a60a9f5
A		 1249 {
1250 data = OSData::withCapacity(16);
1251 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootSignatureKey);
1252 if (sym && data)
1253 {
1254 char c;
0c530ab8 1255 uint8_t value = 0;
3a60a9f5
A		 1256 for (uint32_t i = 0; (c = gIOHibernateBootSignature[i]); i++)
1257 {
1258 if (c >= 'a')
1259 c -= 'a' - 10;
1260 else if (c >= 'A')
1261 c -= 'A' - 10;
1262 else if (c >= '0')
1263 c -= '0';
1264 else
1265 continue;
1266 value = (value << 4) | c;
1267 if (i & 1)
1268 data->appendBytes(&value, sizeof(value));
1269 }
1270 gIOOptionsEntry->setProperty(sym, data);
1271 }
1272 if (sym)
1273 sym->release();
1274 if (data)
1275 data->release();
1276 }
1277 }
3a60a9f5
A		 1278 if (!vars->haveFastBoot)
1279 {
1280 // set boot volume to zero
1281 IODTPlatformExpert * platform = OSDynamicCast(IODTPlatformExpert, IOService::getPlatform());
1282 if (platform && (kIOReturnSuccess == platform->readXPRAM(kXPRamAudioVolume,
1283 &vars->saveBootAudioVolume, sizeof(vars->saveBootAudioVolume))))
1284 {
1285 uint8_t newVolume;
1286 newVolume = vars->saveBootAudioVolume & 0xf8;
1287 platform->writeXPRAM(kXPRamAudioVolume,
1288 &newVolume, sizeof(newVolume));
1289 }
1290 }
b0d623f7 1291#endif /* !i386 && !x86_64 */
3a60a9f5
A		 1292 }
1293 // --
1294
1295 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderSignature;
1296 gIOHibernateState = kIOHibernateStateHibernating;
1297 }
1298 while (false);
1299
1300 return (err);
1301}
1302
1303/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1304
2d21ac55
A		 1305DECLARE_IOHIBERNATEPROGRESSALPHA
1306
1307static void
1308ProgressInit(hibernate_graphics_t * display, uint8_t * screen, uint8_t * saveunder, uint32_t savelen)
1309{
1310 uint32_t rowBytes, pixelShift;
1311 uint32_t x, y;
1312 int32_t blob;
1313 uint32_t alpha, in, color, result;
1314 uint8_t * out;
1315 uint32_t saveindex[kIOHibernateProgressCount] = { 0 };
1316
1317 rowBytes = display->rowBytes;
1318 pixelShift = display->depth >> 4;
1319 if (pixelShift < 1) return;
1320
1321 screen += ((display->width
1322 - kIOHibernateProgressCount * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << (pixelShift - 1))
1323 + (display->height - kIOHibernateProgressOriginY - kIOHibernateProgressHeight) * rowBytes;
1324
1325 for (y = 0; y < kIOHibernateProgressHeight; y++)
1326 {
1327 out = screen + y * rowBytes;
1328 for (blob = 0; blob < kIOHibernateProgressCount; blob++)
1329 {
1330 color = blob ? kIOHibernateProgressDarkGray : kIOHibernateProgressMidGray;
1331 for (x = 0; x < kIOHibernateProgressWidth; x++)
1332 {
1333 alpha = gIOHibernateProgressAlpha[y][x];
1334 result = color;
1335 if (alpha)
1336 {
1337 if (0xff != alpha)
1338 {
1339 if (1 == pixelShift)
1340 {
1341 in = *((uint16_t *)out) & 0x1f; // 16
1342 in = (in << 3) | (in >> 2);
1343 }
1344 else
1345 in = *((uint32_t *)out) & 0xff; // 32
1346 saveunder[blob * kIOHibernateProgressSaveUnderSize + saveindex[blob]++] = in;
1347 result = ((255 - alpha) * in + alpha * result + 0xff) >> 8;
1348 }
1349 if (1 == pixelShift)
1350 {
1351 result >>= 3;
1352 *((uint16_t *)out) = (result << 10) | (result << 5) | result; // 16
1353 }
1354 else
1355 *((uint32_t *)out) = (result << 16) | (result << 8) | result; // 32
1356 }
1357 out += (1 << pixelShift);
1358 }
1359 out += (kIOHibernateProgressSpacing << pixelShift);
1360 }
1361 }
1362}
1363
1364
1365static void
1366ProgressUpdate(hibernate_graphics_t * display, uint8_t * screen, int32_t firstBlob, int32_t select)
1367{
1368 uint32_t rowBytes, pixelShift;
1369 uint32_t x, y;
1370 int32_t blob, lastBlob;
1371 uint32_t alpha, in, color, result;
1372 uint8_t * out;
1373 uint32_t saveindex[kIOHibernateProgressCount] = { 0 };
1374
1375 pixelShift = display->depth >> 4;
1376 if (pixelShift < 1)
1377 return;
1378
1379 rowBytes = display->rowBytes;
1380
1381 screen += ((display->width
1382 - kIOHibernateProgressCount * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << (pixelShift - 1))
1383 + (display->height - kIOHibernateProgressOriginY - kIOHibernateProgressHeight) * rowBytes;
1384
1385 lastBlob = (select < kIOHibernateProgressCount) ? select : (kIOHibernateProgressCount - 1);
1386
1387 screen += (firstBlob * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << pixelShift;
1388
1389 for (y = 0; y < kIOHibernateProgressHeight; y++)
1390 {
1391 out = screen + y * rowBytes;
1392 for (blob = firstBlob; blob <= lastBlob; blob++)
1393 {
1394 color = (blob < select) ? kIOHibernateProgressLightGray : kIOHibernateProgressMidGray;
1395 for (x = 0; x < kIOHibernateProgressWidth; x++)
1396 {
1397 alpha = gIOHibernateProgressAlpha[y][x];
1398 result = color;
1399 if (alpha)
1400 {
1401 if (0xff != alpha)
1402 {
1403 in = display->progressSaveUnder[blob][saveindex[blob]++];
1404 result = ((255 - alpha) * in + alpha * result + 0xff) / 255;
1405 }
1406 if (1 == pixelShift)
1407 {
1408 result >>= 3;
1409 *((uint16_t *)out) = (result << 10) | (result << 5) | result; // 16
1410 }
1411 else
1412 *((uint32_t *)out) = (result << 16) | (result << 8) | result; // 32
1413 }
1414 out += (1 << pixelShift);
1415 }
1416 out += (kIOHibernateProgressSpacing << pixelShift);
1417 }
1418 }
1419}
1420
1421/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1422
3a60a9f5
A		 1423IOReturn
1424IOHibernateSystemHasSlept(void)
1425{
1426 IOHibernateVars * vars = &gIOHibernateVars;
2d21ac55
A		 1427 OSObject * obj;
1428 OSData * data;
1429
1430 obj = IOService::getPMRootDomain()->copyProperty(kIOHibernatePreviewBufferKey);
1431 vars->previewBuffer = OSDynamicCast(IOMemoryDescriptor, obj);
1432 if (obj && !vars->previewBuffer)
1433 obj->release();
3a60a9f5 1434
2d21ac55
A		 1435 vars->consoleMapping = NULL;
1436 if (vars->previewBuffer && (kIOReturnSuccess != vars->previewBuffer->prepare()))
3a60a9f5 1437 {
2d21ac55
A		 1438 vars->previewBuffer->release();
1439 vars->previewBuffer = 0;
1440 }
3a60a9f5 1441
0b4c1975
A		 1442 if ((kIOHibernateOptionProgress & gIOHibernateCurrentHeader->options)
1443 && vars->previewBuffer
1444 && (data = OSDynamicCast(OSData,
2d21ac55
A		 1445 IOService::getPMRootDomain()->getProperty(kIOHibernatePreviewActiveKey))))
1446 {
1447 UInt32 flags = *((UInt32 *)data->getBytesNoCopy());
b0d623f7 1448 HIBPRINT("kIOHibernatePreviewActiveKey %08lx\n", (long)flags);
2d21ac55
A		 1449
1450 IOService::getPMRootDomain()->removeProperty(kIOHibernatePreviewActiveKey);
1451
1452 if (kIOHibernatePreviewUpdates & flags)
1453 {
1454 PE_Video consoleInfo;
1455 hibernate_graphics_t * graphicsInfo = gIOHibernateGraphicsInfo;
1456
1457 IOService::getPlatform()->getConsoleInfo(&consoleInfo);
1458
1459 graphicsInfo->width = consoleInfo.v_width;
1460 graphicsInfo->height = consoleInfo.v_height;
1461 graphicsInfo->rowBytes = consoleInfo.v_rowBytes;
1462 graphicsInfo->depth = consoleInfo.v_depth;
1463 vars->consoleMapping = (uint8_t *) consoleInfo.v_baseAddr;
1464
1465 HIBPRINT("video %p %d %d %d\n",
0b4c1975
A		 1466 vars->consoleMapping, graphicsInfo->depth,
1467 graphicsInfo->width, graphicsInfo->height);
2d21ac55
A		 1468 if (vars->consoleMapping)
1469 ProgressInit(graphicsInfo, vars->consoleMapping,
1470 &graphicsInfo->progressSaveUnder[0][0], sizeof(graphicsInfo->progressSaveUnder));
1471 }
3a60a9f5 1472 }
2d21ac55 1473
3a60a9f5
A		 1474 if (gIOOptionsEntry)
1475 gIOOptionsEntry->sync();
1476
1477 return (kIOReturnSuccess);
1478}
1479
1480/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1481
1482IOReturn
1483IOHibernateSystemWake(void)
1484{
1485 IOHibernateVars * vars = &gIOHibernateVars;
1486
1487 hibernate_teardown(vars->page_list, vars->page_list_wired);
1488
1489 if (vars->videoMapping)
1490 {
1491 if (vars->videoMapSize)
1492 // remove mappings
1493 IOUnmapPages(kernel_map, vars->videoMapping, vars->videoMapSize);
1494 if (vars->videoAllocSize)
1495 // dealloc range
b0d623f7 1496 kmem_free(kernel_map, trunc_page(vars->videoMapping), vars->videoAllocSize);
3a60a9f5
A		 1497 }
1498
1499 if (vars->previewBuffer)
1500 {
1501 vars->previewBuffer->release();
1502 vars->previewBuffer = 0;
1503 }
1504
0b4c1975
A		 1505 if (kIOHibernateStateWakingFromHibernate == gIOHibernateState)
1506 {
1507 IOService::getPMRootDomain()->setProperty(kIOHibernateOptionsKey,
1508 gIOHibernateCurrentHeader->options, 32);
1509 }
1510 else
1511 {
1512 IOService::getPMRootDomain()->removeProperty(kIOHibernateOptionsKey);
1513 }
1514
1515 if ((kIOHibernateStateWakingFromHibernate == gIOHibernateState)
1516 && (kIOHibernateGfxStatusUnknown != gIOHibernateGraphicsInfo->gfxStatus))
1517 {
1518 IOService::getPMRootDomain()->setProperty(kIOHibernateGfxStatusKey,
1519 &gIOHibernateGraphicsInfo->gfxStatus,
1520 sizeof(gIOHibernateGraphicsInfo->gfxStatus));
1521 }
1522 else
1523 {
1524 IOService::getPMRootDomain()->removeProperty(kIOHibernateGfxStatusKey);
1525 }
1526
1527
3a60a9f5
A		 1528 if (vars->fileVars)
1529 {
1530 IOPolledFileClose(vars->fileVars);
1531 }
1532
1533 // invalidate nvram properties - (gIOOptionsEntry != 0) => nvram was touched
1534
0c530ab8 1535#ifdef __ppc__
3a60a9f5
A		 1536 OSData * data = OSData::withCapacity(4);
1537 if (gIOOptionsEntry && data)
1538 {
1539 const OSSymbol * sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKey);
1540 if (sym)
1541 {
1542 gIOOptionsEntry->setProperty(sym, data);
1543 sym->release();
1544 }
1545 sym = OSSymbol::withCStringNoCopy(kIOSelectedBootDeviceKey);
1546 if (sym)
1547 {
1548 if (vars->saveBootDevice)
1549 {
1550 gIOOptionsEntry->setProperty(sym, vars->saveBootDevice);
1551 vars->saveBootDevice->release();
1552 }
1553 sym->release();
1554 }
1555 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKeyKey);
1556 if (sym)
1557 {
1558 gIOOptionsEntry->setProperty(sym, data);
1559 sym->release();
1560 }
1561 sym = OSSymbol::withCStringNoCopy(kIOHibernateMemorySignatureEnvKey);
1562 if (sym)
1563 {
1564 gIOOptionsEntry->removeProperty(sym);
1565 sym->release();
1566 }
1567 }
1568 if (data)
1569 data->release();
1570
1571 if (gIOOptionsEntry)
1572 {
1573 if (!vars->haveFastBoot)
1574 {
1575 // reset boot audio volume
1576 IODTPlatformExpert * platform = OSDynamicCast(IODTPlatformExpert, IOService::getPlatform());
1577 if (platform)
1578 platform->writeXPRAM(kXPRamAudioVolume,
1579 &vars->saveBootAudioVolume, sizeof(vars->saveBootAudioVolume));
1580 }
1581
1582 // sync now to hardware if the booter has not
1583 if (kIOHibernateStateInactive == gIOHibernateState)
1584 gIOOptionsEntry->sync();
1585 else
1586 // just sync the variables in case a later panic syncs nvram (it won't sync variables)
1587 gIOOptionsEntry->syncOFVariables();
1588 }
0c530ab8
A		 1589#endif
1590
b0d623f7 1591#if defined(__i386__) || defined(__x86_64__)
2d21ac55
A		 1592 IOService::getPMRootDomain()->removeProperty(kIOHibernateRTCVariablesKey);
1593
1594 /*
1595 * Hibernate variable is written to NVRAM on platforms in which RtcRam
1596 * is not backed by coin cell. Remove Hibernate data from NVRAM.
1597 */
1598 if (gIOOptionsEntry) {
1599 const OSSymbol * sym = OSSymbol::withCStringNoCopy(kIOHibernateRTCVariablesKey);
1600
1601 if (sym) {
4a3eedf9
A		 1602 if (gIOOptionsEntry->getProperty(sym)) {
1603 gIOOptionsEntry->removeProperty(sym);
1604 gIOOptionsEntry->sync();
1605 }
2d21ac55
A		 1606 sym->release();
1607 }
1608 }
0c530ab8 1609#endif
3a60a9f5
A		 1610
1611 if (vars->srcBuffer)
1612 vars->srcBuffer->release();
1613 if (vars->ioBuffer)
1614 vars->ioBuffer->release();
1615 if (vars->fileExtents)
1616 vars->fileExtents->release();
1617
1618 bzero(vars, sizeof(*vars));
1619
1620// gIOHibernateState = kIOHibernateStateInactive; // leave it for post wake code to see
1621
1622 return (kIOReturnSuccess);
1623}
1624
1625IOReturn
1626IOHibernateSystemPostWake(void)
1627{
1628 if (gIOHibernateFileRef)
1629 {
0c530ab8
A		 1630 // invalidate the image file
1631 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderInvalidSignature;
1632 int err = kern_write_file(gIOHibernateFileRef, 0,
1633 (caddr_t) gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
1634 if (KERN_SUCCESS != err)
1635 HIBLOG("kern_write_file(%d)\n", err);
1636
3a60a9f5
A		 1637 kern_close_file_for_direct_io(gIOHibernateFileRef);
1638 gIOHibernateFileRef = 0;
1639 }
1640 return (kIOReturnSuccess);
1641}
1642
1643/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1644
2d21ac55
A		 1645SYSCTL_STRING(_kern, OID_AUTO, hibernatefile,
1646 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1647 gIOHibernateFilename, sizeof(gIOHibernateFilename), "");
1648SYSCTL_STRING(_kern, OID_AUTO, bootsignature,
1649 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1650 gIOHibernateBootSignature, sizeof(gIOHibernateBootSignature), "");
1651SYSCTL_UINT(_kern, OID_AUTO, hibernatemode,
1652 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN,
1653 &gIOHibernateMode, 0, "");
1654
3a60a9f5
A		 1655void
1656IOHibernateSystemInit(IOPMrootDomain * rootDomain)
1657{
1658 OSData * data = OSData::withBytesNoCopy(&gIOHibernateState, sizeof(gIOHibernateState));
1659 if (data)
1660 {
1661 rootDomain->setProperty(kIOHibernateStateKey, data);
1662 data->release();
1663 }
1664
2d21ac55 1665 if (PE_parse_boot_argn("hfile", gIOHibernateFilename, sizeof(gIOHibernateFilename)))
3a60a9f5
A		 1666 gIOHibernateMode = kIOHibernateModeOn;
1667 else
1668 gIOHibernateFilename[0] = 0;
1669
3a60a9f5 1670 sysctl_register_oid(&sysctl__kern_hibernatefile);
3a60a9f5 1671 sysctl_register_oid(&sysctl__kern_bootsignature);
3a60a9f5
A		 1672 sysctl_register_oid(&sysctl__kern_hibernatemode);
1673}
1674
2d21ac55 1675
3a60a9f5
A		 1676/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1677
1678static void
1679hibernate_setup_for_wake(void)
1680{
1681#if __ppc__
1682 // go slow (state needed for wake)
1683 ml_set_processor_speed(1);
1684#endif
1685}
1686
1687/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1688
0c530ab8
A		 1689#define C_ASSERT(e) typedef char __C_ASSERT__[(e) ? 1 : -1]
1690
0b4c1975
A		 1691static bool
1692no_encrypt_page(vm_offset_t ppnum)
1693{
1694 if (pmap_is_noencrypt((ppnum_t)ppnum) == TRUE)
1695 {
1696 return true;
1697 }
1698 return false;
1699}
1700
1701uint32_t wired_pages_encrypted = 0;
1702uint32_t dirty_pages_encrypted = 0;
1703uint32_t wired_pages_clear = 0;
1704
2d21ac55 1705extern "C" uint32_t
3a60a9f5
A		 1706hibernate_write_image(void)
1707{
1708 IOHibernateImageHeader * header = gIOHibernateCurrentHeader;
1709 IOHibernateVars * vars = &gIOHibernateVars;
1710 IOPolledFileExtent * fileExtents;
1711
0c530ab8
A		 1712 C_ASSERT(sizeof(IOHibernateImageHeader) == 512);
1713
3a60a9f5
A		 1714 uint32_t pageCount, pagesDone;
1715 IOReturn err;
0b4c1975
A		 1716 vm_offset_t ppnum, page;
1717 IOItemCount count;
3a60a9f5
A		 1718 uint8_t * src;
1719 uint8_t * data;
1720 IOByteCount pageCompressedSize;
1721 uint64_t compressedSize, uncompressedSize;
1722 uint64_t image1Size = 0;
1723 uint32_t bitmap_size;
0b4c1975 1724 bool iterDone, pollerOpen, needEncrypt;
3a60a9f5
A		 1725 uint32_t restore1Sum, sum, sum1, sum2;
1726 uint32_t tag;
1727 uint32_t pageType;
1728 uint32_t pageAndCount[2];
1729
1730 AbsoluteTime startTime, endTime;
1731 AbsoluteTime allTime, compTime, decoTime;
0b4c1975 1732 uint64_t compBytes;
3a60a9f5
A		 1733 uint64_t nsec;
1734 uint32_t lastProgressStamp = 0;
1735 uint32_t progressStamp;
2d21ac55 1736 uint32_t blob, lastBlob = (uint32_t) -1L;
3a60a9f5
A		 1737
1738 hibernate_cryptvars_t _cryptvars;
1739 hibernate_cryptvars_t * cryptvars = 0;
1740
0b4c1975
A		 1741 wired_pages_encrypted = 0;
1742 dirty_pages_encrypted = 0;
1743 wired_pages_clear = 0;
1744
3a60a9f5
A		 1745 if (!vars->fileVars || !vars->fileVars->pollers || !vars->fileExtents)
1746 return (false /* sleep */ );
1747
0b4c1975
A		 1748 if (kIOHibernateModeSleep & gIOHibernateMode)
1749 kdebug_enable = save_kdebug_enable;
1750
1751 KERNEL_DEBUG_CONSTANT(IOKDBG_CODE(DBG_HIBERNATE, 1) | DBG_FUNC_START, 0, 0, 0, 0, 0);
1752
3a60a9f5
A		 1753 restore1Sum = sum1 = sum2 = 0;
1754
2d21ac55 1755#if CRYPTO
3a60a9f5
A		 1756 // encryption data. "iv" is the "initial vector".
1757 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1758 {
1759 static const unsigned char first_iv[AES_BLOCK_SIZE]
1760 = { 0xa3, 0x63, 0x65, 0xa9, 0x0b, 0x71, 0x7b, 0x1c,
1761 0xdf, 0x9e, 0x5f, 0x32, 0xd7, 0x61, 0x63, 0xda };
1762
1763 cryptvars = &gIOHibernateCryptWakeContext;
1764 bzero(cryptvars, sizeof(hibernate_cryptvars_t));
1765 aes_encrypt_key(vars->cryptKey,
1766 kIOHibernateAESKeySize,
1767 &cryptvars->ctx.encrypt);
1768 aes_decrypt_key(vars->cryptKey,
1769 kIOHibernateAESKeySize,
1770 &cryptvars->ctx.decrypt);
1771
1772 cryptvars = &_cryptvars;
1773 bzero(cryptvars, sizeof(hibernate_cryptvars_t));
1774 aes_encrypt_key(vars->wiredCryptKey,
1775 kIOHibernateAESKeySize,
1776 &cryptvars->ctx.encrypt);
1777
1778 bcopy(&first_iv[0], &cryptvars->aes_iv[0], AES_BLOCK_SIZE);
1779 bzero(&vars->wiredCryptKey[0], sizeof(vars->wiredCryptKey));
1780 bzero(&vars->cryptKey[0], sizeof(vars->cryptKey));
1781 bzero(gIOHibernateCryptWakeVars, sizeof(hibernate_cryptwakevars_t));
1782 }
2d21ac55 1783#endif /* CRYPTO */
3a60a9f5
A		 1784
1785 hibernate_setup_for_wake();
1786
1787 hibernate_page_list_setall(vars->page_list,
1788 vars->page_list_wired,
1789 &pageCount);
1790
1791 HIBLOG("hibernate_page_list_setall found pageCount %d\n", pageCount);
1792
1793 fileExtents = (IOPolledFileExtent *) vars->fileExtents->getBytesNoCopy();
1794
1795#if 0
1796 count = vars->fileExtents->getLength() / sizeof(IOPolledFileExtent);
1797 for (page = 0; page < count; page++)
1798 {
1799 HIBLOG("fileExtents[%d] %qx, %qx (%qx)\n", page,
1800 fileExtents[page].start, fileExtents[page].length,
1801 fileExtents[page].start + fileExtents[page].length);
1802 }
1803#endif
1804
0b4c1975 1805 needEncrypt = (0 != (kIOHibernateModeEncrypt & gIOHibernateMode));
3a60a9f5
A		 1806
1807 AbsoluteTime_to_scalar(&compTime) = 0;
1808 AbsoluteTime_to_scalar(&decoTime) = 0;
1809
1810 clock_get_uptime(&allTime);
b0d623f7
A		 1811 IOService::getPMRootDomain()->pmStatsRecordEvent(
1812 kIOPMStatsHibernateImageWrite | kIOPMStatsEventStartFlag, allTime);
3a60a9f5
A		 1813
1814 do
1815 {
1816 compressedSize = 0;
1817 uncompressedSize = 0;
3a60a9f5
A		 1818
1819 IOPolledFileSeek(vars->fileVars, sizeof(IOHibernateImageHeader));
1820
1821 HIBLOG("IOHibernatePollerOpen, ml_get_interrupts_enabled %d\n",
1822 ml_get_interrupts_enabled());
1823 err = IOHibernatePollerOpen(vars->fileVars, kIOPolledBeforeSleepState, vars->ioBuffer);
1824 HIBLOG("IOHibernatePollerOpen(%x)\n", err);
1825 pollerOpen = (kIOReturnSuccess == err);
1826 if (!pollerOpen)
1827 break;
1828
1829 // copy file block extent list if larger than header
1830
1831 count = vars->fileExtents->getLength();
1832 if (count > sizeof(header->fileExtentMap))
1833 {
1834 count -= sizeof(header->fileExtentMap);
1835 err = IOPolledFileWrite(vars->fileVars,
1836 ((uint8_t *) &fileExtents[0]) + sizeof(header->fileExtentMap), count, cryptvars);
1837 if (kIOReturnSuccess != err)
1838 break;
1839 }
1840
b0d623f7
A		 1841 uintptr_t hibernateBase;
1842 uintptr_t hibernateEnd;
1843
1844#if defined(__i386__) || defined(__x86_64__)
1845 hibernateBase = sectINITPTB;
1846#else
1847 hibernateBase = sectHIBB;
1848#endif
1849
1850 hibernateEnd = (sectHIBB + sectSizeHIB);
3a60a9f5
A		 1851 // copy out restore1 code
1852
b0d623f7
A		 1853 page = atop_32(hibernateBase);
1854 count = atop_32(round_page(hibernateEnd)) - page;
3a60a9f5
A		 1855 header->restore1CodePage = page;
1856 header->restore1PageCount = count;
b0d623f7
A		 1857 header->restore1CodeOffset = ((uintptr_t) &hibernate_machine_entrypoint) - hibernateBase;
1858 header->restore1StackOffset = ((uintptr_t) &gIOHibernateRestoreStackEnd[0]) - 64 - hibernateBase;
3a60a9f5
A		 1859
1860 // sum __HIB sect, with zeros for the stack
b0d623f7 1861 src = (uint8_t *) trunc_page(hibernateBase);
3a60a9f5
A		 1862 for (page = 0; page < count; page++)
1863 {
1864 if ((src < &gIOHibernateRestoreStack[0]) || (src >= &gIOHibernateRestoreStackEnd[0]))
0b4c1975 1865 restore1Sum += hibernate_sum_page(src, header->restore1CodePage + page);
3a60a9f5 1866 else
0b4c1975 1867 restore1Sum += 0x00000000;
3a60a9f5
A		 1868 src += page_size;
1869 }
1870 sum1 = restore1Sum;
1871
1872 // write the __HIB sect, with zeros for the stack
1873
b0d623f7
A		 1874 src = (uint8_t *) trunc_page(hibernateBase);
1875 count = ((uintptr_t) &gIOHibernateRestoreStack[0]) - trunc_page(hibernateBase);
3a60a9f5
A		 1876 if (count)
1877 {
1878 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1879 if (kIOReturnSuccess != err)
1880 break;
1881 }
1882 err = IOPolledFileWrite(vars->fileVars,
1883 (uint8_t *) 0,
1884 &gIOHibernateRestoreStackEnd[0] - &gIOHibernateRestoreStack[0],
1885 cryptvars);
1886 if (kIOReturnSuccess != err)
1887 break;
1888 src = &gIOHibernateRestoreStackEnd[0];
b0d623f7 1889 count = round_page(hibernateEnd) - ((uintptr_t) src);
3a60a9f5
A		 1890 if (count)
1891 {
1892 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1893 if (kIOReturnSuccess != err)
1894 break;
1895 }
1896
1897 // write the preview buffer
1898
1899 addr64_t phys64;
1900 IOByteCount segLen;
1901
2d21ac55 1902 if (vars->previewBuffer)
3a60a9f5
A		 1903 {
1904 ppnum = 0;
1905 count = 0;
1906 do
1907 {
b0d623f7 1908 phys64 = vars->previewBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone);
3a60a9f5
A		 1909 pageAndCount[0] = atop_64(phys64);
1910 pageAndCount[1] = atop_32(segLen);
1911 err = IOPolledFileWrite(vars->fileVars,
1912 (const uint8_t *) &pageAndCount, sizeof(pageAndCount),
1913 cryptvars);
1914 if (kIOReturnSuccess != err)
1915 break;
1916 count += segLen;
1917 ppnum += sizeof(pageAndCount);
1918 }
1919 while (phys64);
1920 if (kIOReturnSuccess != err)
1921 break;
1922
b0d623f7 1923 src = (uint8_t *) vars->previewBuffer->getPhysicalSegment(0, NULL, _kIOMemorySourceSegment);
2d21ac55 1924 count = vars->previewBuffer->getLength();
3a60a9f5
A		 1925
1926 header->previewPageListSize = ppnum;
1927 header->previewSize = count + ppnum;
1928
1929 for (page = 0; page < count; page += page_size)
0b4c1975
A		 1930 {
1931 phys64 = vars->previewBuffer->getPhysicalSegment(page, NULL, kIOMemoryMapperNone);
1932 sum1 += hibernate_sum_page(src + page, atop_64(phys64));
1933 }
3a60a9f5
A		 1934 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
1935 if (kIOReturnSuccess != err)
1936 break;
1937 }
1938
1939 // mark areas for no save
1940
1941 for (count = 0;
b0d623f7 1942 (phys64 = vars->ioBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 1943 count += segLen)
1944 {
1945 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1946 atop_64(phys64), atop_32(segLen),
1947 kIOHibernatePageStateFree);
1948 pageCount -= atop_32(segLen);
1949 }
1950
1951 for (count = 0;
b0d623f7 1952 (phys64 = vars->srcBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 1953 count += segLen)
1954 {
1955 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1956 atop_64(phys64), atop_32(segLen),
1957 kIOHibernatePageStateFree);
1958 pageCount -= atop_32(segLen);
1959 }
1960
1961 // copy out bitmap of pages available for trashing during restore
1962
1963 bitmap_size = vars->page_list_wired->list_size;
1964 src = (uint8_t *) vars->page_list_wired;
1965 err = IOPolledFileWrite(vars->fileVars, src, bitmap_size, cryptvars);
1966 if (kIOReturnSuccess != err)
1967 break;
1968
1969 // mark more areas for no save, but these are not available
1970 // for trashing during restore
0c530ab8
A		 1971
1972 hibernate_page_list_set_volatile(vars->page_list, vars->page_list_wired, &pageCount);
3a60a9f5 1973
b0d623f7
A		 1974 page = atop_32(hibernateBase);
1975 count = atop_32(round_page(hibernateEnd)) - page;
3a60a9f5
A		 1976 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1977 page, count,
1978 kIOHibernatePageStateFree);
1979 pageCount -= count;
3a60a9f5
A		 1980
1981 if (vars->previewBuffer) for (count = 0;
b0d623f7 1982 (phys64 = vars->previewBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 1983 count += segLen)
1984 {
1985 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
1986 atop_64(phys64), atop_32(segLen),
1987 kIOHibernatePageStateFree);
1988 pageCount -= atop_32(segLen);
1989 }
1990
1991 src = (uint8_t *) vars->srcBuffer->getBytesNoCopy();
1992
0c530ab8 1993 pagesDone = 0;
2d21ac55 1994 lastBlob = 0;
3a60a9f5
A		 1995
1996 HIBLOG("writing %d pages\n", pageCount);
1997
0b4c1975
A		 1998 enum
1999 // pageType
2000 {
2001 kWired = 0x02,
2002 kEncrypt = 0x01,
2003 kWiredEncrypt = kWired | kEncrypt,
2004 kWiredClear = kWired,
2005 kUnwiredEncrypt = kEncrypt
2006 };
2007
2008 for (pageType = kWiredEncrypt; pageType >= kUnwiredEncrypt; pageType--)
3a60a9f5 2009 {
0b4c1975 2010 if (needEncrypt && (kEncrypt & pageType))
3a60a9f5 2011 {
0b4c1975
A		 2012 vars->fileVars->encryptStart = (vars->fileVars->position & ~(AES_BLOCK_SIZE - 1));
2013 vars->fileVars->encryptEnd = UINT64_MAX;
2014 HIBLOG("encryptStart %qx\n", vars->fileVars->encryptStart);
2015
2016 if (kUnwiredEncrypt == pageType)
3a60a9f5 2017 {
0b4c1975
A		 2018 // start unwired image
2019 bcopy(&cryptvars->aes_iv[0],
2020 &gIOHibernateCryptWakeContext.aes_iv[0],
2021 sizeof(cryptvars->aes_iv));
2022 cryptvars = &gIOHibernateCryptWakeContext;
3a60a9f5 2023 }
0b4c1975
A		 2024 }
2025 for (iterDone = false, ppnum = 0; !iterDone; )
2026 {
2027 count = hibernate_page_list_iterate((kWired & pageType)
2028 ? vars->page_list_wired : vars->page_list,
2029 &ppnum);
2030// kprintf("[%d](%x : %x)\n", pageType, ppnum, count);
2031 iterDone = !count;
3a60a9f5 2032
0b4c1975 2033 if (count && (kWired & pageType) && needEncrypt)
3a60a9f5 2034 {
0b4c1975
A		 2035 uint32_t checkIndex;
2036 for (checkIndex = 0;
2037 (checkIndex < count)
2038 && (((kEncrypt & pageType) == 0) == no_encrypt_page(ppnum + checkIndex));
2039 checkIndex++)
2040 {}
2041 if (!checkIndex)
2042 {
2043 ppnum++;
2044 continue;
2045 }
2046 count = checkIndex;
3a60a9f5
A		 2047 }
2048
0b4c1975
A		 2049 switch (pageType)
2050 {
2051 case kWiredEncrypt: wired_pages_encrypted += count; break;
2052 case kWiredClear: wired_pages_clear += count; break;
2053 case kUnwiredEncrypt: dirty_pages_encrypted += count; break;
2054 }
2055
2056 if (iterDone && (kWiredEncrypt == pageType)) {/* not yet end of wired list */}
3a60a9f5 2057 else
3a60a9f5 2058 {
0b4c1975
A		 2059 pageAndCount[0] = ppnum;
2060 pageAndCount[1] = count;
2061 err = IOPolledFileWrite(vars->fileVars,
2062 (const uint8_t *) &pageAndCount, sizeof(pageAndCount),
2063 cryptvars);
2064 if (kIOReturnSuccess != err)
2065 break;
3a60a9f5 2066 }
0b4c1975
A		 2067
2068 for (page = ppnum; page < (ppnum + count); page++)
3a60a9f5 2069 {
0b4c1975
A		 2070 err = IOMemoryDescriptorWriteFromPhysical(vars->srcBuffer, 0, ptoa_64(page), page_size);
2071 if (err)
2072 {
2073 HIBLOG("IOMemoryDescriptorWriteFromPhysical %d [%ld] %x\n", __LINE__, (long)page, err);
2074 break;
2075 }
2076
2077 sum = hibernate_sum_page(src, page);
2078 if (kWired & pageType)
2079 sum1 += sum;
2080 else
2081 sum2 += sum;
2082
2083 clock_get_uptime(&startTime);
2084
2085 pageCompressedSize = WKdm_compress ((WK_word*) src, (WK_word*) (src + page_size), PAGE_SIZE_IN_WORDS);
2086
3a60a9f5 2087 clock_get_uptime(&endTime);
0b4c1975
A		 2088 ADD_ABSOLUTETIME(&compTime, &endTime);
2089 SUB_ABSOLUTETIME(&compTime, &startTime);
2090 compBytes += page_size;
2091
2092 if (kIOHibernateModeEncrypt & gIOHibernateMode)
2093 pageCompressedSize = (pageCompressedSize + AES_BLOCK_SIZE - 1) & ~(AES_BLOCK_SIZE - 1);
2094
2095 if (pageCompressedSize > page_size)
3a60a9f5 2096 {
0b4c1975
A		 2097// HIBLOG("------------lose: %d\n", pageCompressedSize);
2098 pageCompressedSize = page_size;
2099 }
2100
2101 if (pageCompressedSize != page_size)
2102 data = (src + page_size);
2103 else
2104 data = src;
2105
2106 tag = pageCompressedSize | kIOHibernateTagSignature;
2107 err = IOPolledFileWrite(vars->fileVars, (const uint8_t *) &tag, sizeof(tag), cryptvars);
2108 if (kIOReturnSuccess != err)
2109 break;
2110
2111 err = IOPolledFileWrite(vars->fileVars, data, (pageCompressedSize + 3) & ~3, cryptvars);
2112 if (kIOReturnSuccess != err)
2113 break;
2114
2115 compressedSize += pageCompressedSize;
2116 if (pageCompressedSize)
2117 uncompressedSize += page_size;
2118 pagesDone++;
2119
2120 if (vars->consoleMapping && (0 == (1023 & pagesDone)))
2121 {
2122 blob = ((pagesDone * kIOHibernateProgressCount) / pageCount);
2123 if (blob != lastBlob)
2124 {
2125 ProgressUpdate(gIOHibernateGraphicsInfo, vars->consoleMapping, lastBlob, blob);
2126 lastBlob = blob;
2127 }
2128 }
2129 if (0 == (8191 & pagesDone))
2130 {
2131 clock_get_uptime(&endTime);
2132 SUB_ABSOLUTETIME(&endTime, &allTime);
2133 absolutetime_to_nanoseconds(endTime, &nsec);
2134 progressStamp = nsec / 750000000ULL;
2135 if (progressStamp != lastProgressStamp)
2136 {
2137 lastProgressStamp = progressStamp;
2138 HIBPRINT("pages %d (%d%%)\n", pagesDone, (100 * pagesDone) / pageCount);
2139 }
3a60a9f5
A		 2140 }
2141 }
0b4c1975
A		 2142 if (kIOReturnSuccess != err)
2143 break;
2144 ppnum = page;
3a60a9f5 2145 }
0b4c1975 2146
3a60a9f5
A		 2147 if (kIOReturnSuccess != err)
2148 break;
0b4c1975
A		 2149
2150 if ((kEncrypt & pageType))
3a60a9f5 2151 {
0b4c1975
A		 2152 vars->fileVars->encryptEnd = (vars->fileVars->position + AES_BLOCK_SIZE - 1)
2153 & ~(AES_BLOCK_SIZE - 1);
2154 HIBLOG("encryptEnd %qx\n", vars->fileVars->encryptEnd);
2155 }
2156
2157 if (kWiredEncrypt != pageType)
2158 {
2159 // end of image1/2 - fill to next block
3a60a9f5
A		 2160 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
2161 if (kIOReturnSuccess != err)
2162 break;
0b4c1975
A		 2163 }
2164 if (kWiredClear == pageType)
2165 {
2166 // end wired image
2167 header->encryptStart = vars->fileVars->encryptStart;
2168 header->encryptEnd = vars->fileVars->encryptEnd;
3a60a9f5 2169 image1Size = vars->fileVars->position;
0b4c1975
A		 2170 HIBLOG("image1Size %qd, encryptStart1 %qx, End1 %qx\n",
2171 image1Size, header->encryptStart, header->encryptEnd);
3a60a9f5
A		 2172 }
2173 }
3a60a9f5
A		 2174 if (kIOReturnSuccess != err)
2175 break;
2176
2177 // Header:
2178
2179 header->imageSize = vars->fileVars->position;
2180 header->image1Size = image1Size;
2181 header->bitmapSize = bitmap_size;
2182 header->pageCount = pageCount;
3a60a9f5
A		 2183
2184 header->restore1Sum = restore1Sum;
2185 header->image1Sum = sum1;
2186 header->image2Sum = sum2;
2187
2188 count = vars->fileExtents->getLength();
2189 if (count > sizeof(header->fileExtentMap))
2190 {
2191 header->fileExtentMapSize = count;
2192 count = sizeof(header->fileExtentMap);
2193 }
2194 else
2195 header->fileExtentMapSize = sizeof(header->fileExtentMap);
2196 bcopy(&fileExtents[0], &header->fileExtentMap[0], count);
2197
0b4c1975
A		 2198 header->deviceBase = vars->fileVars->block0;
2199
3a60a9f5
A		 2200 IOPolledFileSeek(vars->fileVars, 0);
2201 err = IOPolledFileWrite(vars->fileVars,
2202 (uint8_t *) header, sizeof(IOHibernateImageHeader),
2203 cryptvars);
2204 if (kIOReturnSuccess != err)
2205 break;
2206 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
2207 if (kIOReturnSuccess != err)
2208 break;
2d21ac55 2209 err = IOHibernatePollerIODone(vars->fileVars, true);
3a60a9f5
A		 2210 if (kIOReturnSuccess != err)
2211 break;
2212 }
2213 while (false);
2214
2215 clock_get_uptime(&endTime);
b0d623f7
A		 2216
2217 IOService::getPMRootDomain()->pmStatsRecordEvent(
2218 kIOPMStatsHibernateImageWrite | kIOPMStatsEventStopFlag, endTime);
2219
3a60a9f5
A		 2220 SUB_ABSOLUTETIME(&endTime, &allTime);
2221 absolutetime_to_nanoseconds(endTime, &nsec);
2222 HIBLOG("all time: %qd ms, ",
2223 nsec / 1000000ULL);
2224
2225 absolutetime_to_nanoseconds(compTime, &nsec);
2226 HIBLOG("comp time: %qd ms, ",
2227 nsec / 1000000ULL);
2228
2229 absolutetime_to_nanoseconds(decoTime, &nsec);
2230 HIBLOG("deco time: %qd ms, ",
2231 nsec / 1000000ULL);
2232
2233 HIBLOG("\nimage %qd, uncompressed %qd (%d), compressed %qd (%d%%), sum1 %x, sum2 %x\n",
2234 header->imageSize,
2235 uncompressedSize, atop_32(uncompressedSize), compressedSize,
8f6c56a5 2236 uncompressedSize ? ((int) ((compressedSize * 100ULL) / uncompressedSize)) : 0,
3a60a9f5
A		 2237 sum1, sum2);
2238
0b4c1975
A		 2239 HIBLOG("wired_pages_encrypted %d, wired_pages_clear %d, dirty_pages_encrypted %d\n",
2240 wired_pages_encrypted, wired_pages_clear, dirty_pages_encrypted);
2241
c910b4d9
A		 2242 if (vars->fileVars->io)
2243 (void) IOHibernatePollerIODone(vars->fileVars, false);
2244
3a60a9f5
A		 2245 if (pollerOpen)
2246 IOHibernatePollerClose(vars->fileVars, kIOPolledBeforeSleepState);
2247
2d21ac55
A		 2248 if (vars->consoleMapping)
2249 ProgressUpdate(gIOHibernateGraphicsInfo,
2250 vars->consoleMapping, 0, kIOHibernateProgressCount);
2251
3a60a9f5
A		 2252 HIBLOG("hibernate_write_image done(%x)\n", err);
2253
2254 // should we come back via regular wake, set the state in memory.
2255 gIOHibernateState = kIOHibernateStateInactive;
2256
0b4c1975
A		 2257 KERNEL_DEBUG_CONSTANT(IOKDBG_CODE(DBG_HIBERNATE, 1) | DBG_FUNC_END,
2258 wired_pages_encrypted, wired_pages_clear, dirty_pages_encrypted, 0, 0);
2259
2d21ac55
A		 2260 if (kIOReturnSuccess == err)
2261 {
2262 if (kIOHibernateModeSleep & gIOHibernateMode)
2263 {
2264 return (kIOHibernatePostWriteSleep);
2265 }
2266 else if(kIOHibernateModeRestart & gIOHibernateMode)
2267 {
2268 return (kIOHibernatePostWriteRestart);
2269 }
2270 else
2271 {
2272 /* by default, power down */
2273 return (kIOHibernatePostWriteHalt);
2274 }
2275 }
2276 else if (kIOReturnAborted == err)
2277 {
2278 return (kIOHibernatePostWriteWake);
2279 }
3a60a9f5 2280 else
3a60a9f5 2281 {
2d21ac55
A		 2282 /* on error, sleep */
2283 return (kIOHibernatePostWriteSleep);
3a60a9f5
A		 2284 }
2285}
2286
2287/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
2288
2289extern "C" void
2290hibernate_machine_init(void)
2291{
2292 IOReturn err;
2293 uint32_t sum;
2294 uint32_t pagesDone;
0b4c1975 2295 uint32_t pagesRead = 0;
3a60a9f5
A		 2296 AbsoluteTime allTime, endTime;
2297 uint64_t nsec;
2298 uint32_t lastProgressStamp = 0;
2299 uint32_t progressStamp;
2300 uint64_t progressZeroPosition = 0;
2301 uint32_t blob, lastBlob = (uint32_t) -1L;
2302 hibernate_cryptvars_t * cryptvars = 0;
2303
2304 IOHibernateVars * vars = &gIOHibernateVars;
2305
2306 if (!vars->fileVars || !vars->fileVars->pollers || !vars->fileExtents)
2307 return;
2308
3a60a9f5
A		 2309 sum = gIOHibernateCurrentHeader->actualImage1Sum;
2310 pagesDone = gIOHibernateCurrentHeader->actualUncompressedPages;
2311
2312 HIBLOG("hibernate_machine_init: state %d, image pages %d, sum was %x, image1Size %qx, conflictCount %d, nextFree %x\n",
2313 gIOHibernateState, pagesDone, sum, gIOHibernateCurrentHeader->image1Size,
2314 gIOHibernateCurrentHeader->conflictCount, gIOHibernateCurrentHeader->nextFree);
2315
2316 if (kIOHibernateStateWakingFromHibernate != gIOHibernateState)
2317 {
2318 HIBLOG("regular wake\n");
2319 return;
2320 }
2321
2322 HIBPRINT("diag %x %x %x %x\n",
2323 gIOHibernateCurrentHeader->diag[0], gIOHibernateCurrentHeader->diag[1],
2324 gIOHibernateCurrentHeader->diag[2], gIOHibernateCurrentHeader->diag[3]);
2325
0b4c1975 2326 HIBPRINT("video %x %d %d %d status %x\n",
3a60a9f5 2327 gIOHibernateGraphicsInfo->physicalAddress, gIOHibernateGraphicsInfo->depth,
0b4c1975 2328 gIOHibernateGraphicsInfo->width, gIOHibernateGraphicsInfo->height, gIOHibernateGraphicsInfo->gfxStatus);
3a60a9f5 2329
0c530ab8
A		 2330 if ((kIOHibernateModeDiscardCleanActive | kIOHibernateModeDiscardCleanInactive) & gIOHibernateMode)
2331 hibernate_page_list_discard(vars->page_list);
2332
2333 boot_args *args = (boot_args *) PE_state.bootArgs;
2334
2335 if (vars->videoMapping
2336 && gIOHibernateGraphicsInfo->physicalAddress
2337 && (args->Video.v_baseAddr == gIOHibernateGraphicsInfo->physicalAddress))
3a60a9f5
A		 2338 {
2339 vars->videoMapSize = round_page(gIOHibernateGraphicsInfo->height
2340 * gIOHibernateGraphicsInfo->rowBytes);
2341 IOMapPages(kernel_map,
2342 vars->videoMapping, gIOHibernateGraphicsInfo->physicalAddress,
2343 vars->videoMapSize, kIOMapInhibitCache );
2344 }
2345
0c530ab8
A		 2346 uint8_t * src = (uint8_t *) vars->srcBuffer->getBytesNoCopy();
2347
2348 if (gIOHibernateWakeMapSize)
2349 {
2350 err = IOMemoryDescriptorWriteFromPhysical(vars->srcBuffer, 0, ptoa_64(gIOHibernateWakeMap),
2351 gIOHibernateWakeMapSize);
2352 if (kIOReturnSuccess == err)
2353 hibernate_newruntime_map(src, gIOHibernateWakeMapSize,
2354 gIOHibernateCurrentHeader->systemTableOffset);
2355 gIOHibernateWakeMap = 0;
2356 gIOHibernateWakeMapSize = 0;
2357 }
2358
3a60a9f5
A		 2359 uint32_t decoOffset;
2360
2361 clock_get_uptime(&allTime);
2362
2363 HIBLOG("IOHibernatePollerOpen(), ml_get_interrupts_enabled %d\n", ml_get_interrupts_enabled());
2364 err = IOHibernatePollerOpen(vars->fileVars, kIOPolledAfterSleepState, 0);
2365 HIBLOG("IOHibernatePollerOpen(%x)\n", err);
2366
2367 if (gIOHibernateCurrentHeader->previewSize)
2368 progressZeroPosition = gIOHibernateCurrentHeader->previewSize
2369 + gIOHibernateCurrentHeader->fileExtentMapSize
2370 - sizeof(gIOHibernateCurrentHeader->fileExtentMap)
2371 + ptoa_64(gIOHibernateCurrentHeader->restore1PageCount);
2372
2373 IOPolledFileSeek(vars->fileVars, gIOHibernateCurrentHeader->image1Size);
2374
2d21ac55 2375 if (vars->videoMapSize)
3a60a9f5
A		 2376 {
2377 lastBlob = ((vars->fileVars->position - progressZeroPosition) * kIOHibernateProgressCount)
2378 / (gIOHibernateCurrentHeader->imageSize - progressZeroPosition);
2379 ProgressUpdate(gIOHibernateGraphicsInfo, (uint8_t *) vars->videoMapping, 0, lastBlob);
2380 }
2381
2382 cryptvars = (kIOHibernateModeEncrypt & gIOHibernateMode) ? &gIOHibernateCryptWakeContext : 0;
2383 if (kIOHibernateModeEncrypt & gIOHibernateMode)
2384 {
2385 cryptvars = &gIOHibernateCryptWakeContext;
2386 bcopy(&gIOHibernateCryptWakeVars->aes_iv[0],
2387 &cryptvars->aes_iv[0],
2388 sizeof(cryptvars->aes_iv));
2389 }
2390
2391 // kick off the read ahead
2392 vars->fileVars->io = false;
2393 vars->fileVars->bufferHalf = 0;
2394 vars->fileVars->bufferLimit = 0;
2395 vars->fileVars->lastRead = 0;
2396 vars->fileVars->bufferOffset = vars->fileVars->bufferLimit;
2397
2398 IOPolledFileRead(vars->fileVars, 0, 0, cryptvars);
2399 vars->fileVars->bufferOffset = vars->fileVars->bufferLimit;
2400 // --
2401
2402 HIBLOG("hibernate_machine_init reading\n");
2403
2404 uint32_t * header = (uint32_t *) src;
2405 sum = 0;
2406
2407 do
2408 {
2409 unsigned int count;
2410 unsigned int page;
2411 uint32_t tag;
2412 vm_offset_t ppnum, compressedSize;
2413
36401178
A		 2414 err = IOPolledFileRead(vars->fileVars, src, 8, cryptvars);
2415 if (kIOReturnSuccess != err)
2416 break;
3a60a9f5
A		 2417
2418 ppnum = header[0];
2419 count = header[1];
2420
2421// HIBPRINT("(%x, %x)\n", ppnum, count);
2422
2423 if (!count)
2424 break;
2425
2426 for (page = 0; page < count; page++)
2427 {
36401178
A		 2428 err = IOPolledFileRead(vars->fileVars, (uint8_t *) &tag, 4, cryptvars);
2429 if (kIOReturnSuccess != err)
2430 break;
3a60a9f5
A		 2431
2432 compressedSize = kIOHibernateTagLength & tag;
36401178
A		 2433 if (kIOHibernateTagSignature != (tag & ~kIOHibernateTagLength))
2434 {
2435 err = kIOReturnIPCError;
2436 break;
2437 }
2438
3a60a9f5
A		 2439 if (!compressedSize)
2440 {
2441 ppnum++;
2442 pagesDone++;
2443 continue;
2444 }
2445
36401178
A		 2446 err = IOPolledFileRead(vars->fileVars, src, (compressedSize + 3) & ~3, cryptvars);
2447 if (kIOReturnSuccess != err)
2448 break;
2449
2450 if (compressedSize < page_size)
3a60a9f5
A		 2451 {
2452 decoOffset = page_size;
2453 WKdm_decompress((WK_word*) src, (WK_word*) (src + decoOffset), PAGE_SIZE_IN_WORDS);
2454 }
2455 else
2456 decoOffset = 0;
2457
0b4c1975 2458 sum += hibernate_sum_page((src + decoOffset), ppnum);
3a60a9f5
A		 2459
2460 err = IOMemoryDescriptorReadToPhysical(vars->srcBuffer, decoOffset, ptoa_64(ppnum), page_size);
2461 if (err)
36401178 2462 {
b0d623f7 2463 HIBLOG("IOMemoryDescriptorReadToPhysical [%ld] %x\n", (long)ppnum, err);
36401178
A		 2464 break;
2465 }
3a60a9f5
A		 2466
2467 ppnum++;
2468 pagesDone++;
0b4c1975 2469 pagesRead++;
3a60a9f5 2470
2d21ac55 2471 if (vars->videoMapSize && (0 == (1023 & pagesDone)))
3a60a9f5
A		 2472 {
2473 blob = ((vars->fileVars->position - progressZeroPosition) * kIOHibernateProgressCount)
2474 / (gIOHibernateCurrentHeader->imageSize - progressZeroPosition);
2475 if (blob != lastBlob)
2476 {
2477 ProgressUpdate(gIOHibernateGraphicsInfo, (uint8_t *) vars->videoMapping, lastBlob, blob);
2478 lastBlob = blob;
2479 }
2480 }
2481
2482 if (0 == (8191 & pagesDone))
2483 {
2484 clock_get_uptime(&endTime);
2485 SUB_ABSOLUTETIME(&endTime, &allTime);
2486 absolutetime_to_nanoseconds(endTime, &nsec);
2487 progressStamp = nsec / 750000000ULL;
2488 if (progressStamp != lastProgressStamp)
2489 {
2490 lastProgressStamp = progressStamp;
2491 HIBPRINT("pages %d (%d%%)\n", pagesDone,
2492 (100 * pagesDone) / gIOHibernateCurrentHeader->pageCount);
2493 }
2494 }
2495 }
2496 }
2497 while (true);
2498
36401178
A		 2499 if (kIOReturnSuccess != err)
2500 panic("Hibernate restore error %x", err);
2501
3a60a9f5
A		 2502 gIOHibernateCurrentHeader->actualImage2Sum = sum;
2503
2504 if (vars->fileVars->io)
2d21ac55 2505 (void) IOHibernatePollerIODone(vars->fileVars, false);
3a60a9f5
A		 2506
2507 err = IOHibernatePollerClose(vars->fileVars, kIOPolledAfterSleepState);
2508
2d21ac55 2509 if (vars->videoMapSize)
3a60a9f5
A		 2510 ProgressUpdate(gIOHibernateGraphicsInfo,
2511 (uint8_t *) vars->videoMapping, 0, kIOHibernateProgressCount);
2512
2513 clock_get_uptime(&endTime);
b0d623f7
A		 2514
2515 IOService::getPMRootDomain()->pmStatsRecordEvent(
2516 kIOPMStatsHibernateImageRead | kIOPMStatsEventStartFlag, allTime);
2517 IOService::getPMRootDomain()->pmStatsRecordEvent(
2518 kIOPMStatsHibernateImageRead | kIOPMStatsEventStopFlag, endTime);
2519
3a60a9f5
A		 2520 SUB_ABSOLUTETIME(&endTime, &allTime);
2521 absolutetime_to_nanoseconds(endTime, &nsec);
2522
2523 HIBLOG("hibernate_machine_init pagesDone %d sum2 %x, time: %qd ms\n",
2524 pagesDone, sum, nsec / 1000000ULL);
0b4c1975
A		 2525
2526 KERNEL_DEBUG_CONSTANT(IOKDBG_CODE(DBG_HIBERNATE, 2) | DBG_FUNC_NONE, pagesRead, pagesDone, 0, 0, 0);
3a60a9f5
A		 2527}
2528
2529/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
mirror of XNU