libkdd/README.md

   1 Kernel Data Descriptors
   2 =======================
   3
   4 This project allows for dynamic data to be passed from the kernel to userspace tools without binding them to particular version of
   5 struct definition. The `libkdd` library provides convenient API for parsing and interpreting `kernel chunked data`.
   6
   7 The libkdd APIs are defined in [kdd.h](./kdd.h)
   8
   9 The `KCDATA` format
  10 ===================
  11
  12 The format for data is setup in a generic format as follows
  13
  14 Layout of data structure
  15 ------------------------
  16
  17         |         8 - bytes         |
  18         |---------------------------|  ------ offset = 00
  19         |  type = MAGIC |  LENGTH   |  # BEGIN Header
  20         |            0              |
  21         |---------------------------|  ------ offset = 16
  22         |      type     |  size     |  # chunk header
  23         |          flags            |
  24         |---------------------------|  ------ offset = 32
  25         |           data            |  # arbitrary data (len=16)
  26         |___________data____________|
  27         |---------------------------|  ------ offset = 48
  28         |      type     |   size    |  # chunk header
  29         |          flags            |
  30         |---------------------------|  ------ offset = 64
  31         |           data            |  # arbitrary data (len=32)
  32         |           data            |
  33         |           data            |
  34         |___________data____________|
  35         |---------------------------|  ------ offset = 96
  36         |  type = END   |  size=0   |  # chunk header
  37         |            0              |
  38
  39
  40 The type field describes what kind of data is passed. For example type = `TASK_CRASHINFO_UUID` means the following data is a uuid.
  41 These types need to be defined in task_corpses.h for easy consumption by userspace inspection tools.
  42
  43 Some range of types is reserved for special types like ints, longs etc. A cool new functionality made possible with this
  44 extensible data format is that kernel can decide to put more information as required without requiring user space tools to
  45 re-compile to be compatible. The case of `rusage` struct versions could be introduced without breaking existing tools.
  46
  47 Feature description: Generic data with description
  48 -------------------
  49 Further more generic data with description is very much possible now. For example
  50
  51         - kcdata_add_uint64_with_description(cdatainfo, 0x700, "NUM MACH PORTS");
  52         - and more functions that allow adding description.
  53
  54 The userspace tools can then look at the description and print the data even if they are not compiled with knowledge of the field apriori.
  55
  56         Example data:
  57         0000  57 f1 ad de 00 00 00 00 00 00 00 00 00 00 00 00  W...............
  58         0010  01 00 00 00 00 00 00 00 30 00 00 00 00 00 00 00  ........0.......
  59         0020  50 49 44 00 00 00 00 00 00 00 00 00 00 00 00 00  PID.............
  60         0030  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  61         0040  9c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  62         0050  01 00 00 00 00 00 00 00 30 00 00 00 00 00 00 00  ........0.......
  63         0060  50 41 52 45 4e 54 20 50 49 44 00 00 00 00 00 00  PARENT PID......
  64         0070  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  65         0080  01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  66         0090  ed 58 91 f1
  67
  68
  69 Feature description: Container markers for compound data
  70 ------------------
  71
  72 If a given kernel data type is complex and requires adding multiple optional fields inside a container
  73 object for a consumer to understand arbitrary data, we package it using container markers.
  74
  75 For example, the stackshot code gathers information and describes the state of a given task with respect
  76 to many subsystems. It includes data such as io stats, vm counters, process names/flags and syscall counts.
  77
  78         kcdata_add_container_marker(kcdata_p, KCDATA_TYPE_CONTAINER_BEGIN, STACKSHOT_KCCONTAINER_TASK, task_uniqueid);
  79         // add multiple data, or add_<type>_with_description()s here
  80
  81         kcdata_add_container_marker(kcdata_p, KCDATA_TYPE_CONTAINER_END, STACKSHOT_KCCONTAINER_TASK, task_uniqueid);
  82
  83
  84 Feature description: Custom Data formats on demand
  85 --------------------
  86
  87 With the self describing nature of format, the kernel provider can describe a data type (uniquely identified by a number) and use
  88 it in the buffer for sending data. The consumer can parse the type information and have knowledge of describing incoming data.
  89 Following is an example of how we can describe a kernel specific struct sample_disk_io_stats in buffer.
  90
  91         struct sample_disk_io_stats {
  92             uint64_t        disk_reads_count;
  93             uint64_t        disk_reads_size;
  94             uint64_t        io_priority_count[4];
  95             uint64_t        io_priority_size;
  96         } __attribute__ ((packed));
  97
  98
  99         struct kcdata_subtype_descriptor disk_io_stats_def[] = {
 100             {KCS_SUBTYPE_FLAGS_NONE, KC_ST_UINT64, 0 * sizeof(uint64_t), sizeof(uint64_t), "disk_reads_count"},
 101             {KCS_SUBTYPE_FLAGS_NONE, KC_ST_UINT64, 1 * sizeof(uint64_t), sizeof(uint64_t), "disk_reads_size"},
 102             {KCS_SUBTYPE_FLAGS_ARRAY, KC_ST_UINT64, 2 * sizeof(uint64_t), KCS_SUBTYPE_PACK_SIZE(4, sizeof(uint64_t)), "io_priority_count"},
 103             {KCS_SUBTYPE_FLAGS_ARRAY, KC_ST_UINT64, (2 + 4) * sizeof(uint64_t), sizeof(uint64_t), "io_priority_size"},
 104         };
 105
 106 Now you can add this custom type definition into the buffer as
 107         kcdata_add_type_definition(kcdata_p, KCTYPE_SAMPLE_DISK_IO_STATS, "sample_disk_io_stats",
 108                  &disk_io_stats_def[0], sizeof(disk_io_stats_def)/sizeof(struct kcdata_subtype_descriptor));
 109