doc/acquire-additional-files.txt

   1 # Acquire additional files in 'update' operations
   2
   3 The download and verification of data from multiple sources in different
   4 compression formats, with partial downloads and patches is an involved
   5 process which is hard to implement correctly and securely.
   6
   7 APT frontends share the code and binaries to make this happen in libapt
   8 with the Acquire system, supported by helpers shipped in the apt package
   9 itself and additional transports in individual packages like
  10 apt-transport-https.
  11
  12 For its own operation libapt needs or can make use of Packages, Sources
  13 and Translation-* files, which it will acquire by default, but
  14 a repository might contain more data files (e.g.  Contents) a frontend
  15 might want to use and would therefore need to be downloaded as well
  16 (e.g. apt-file).
  17
  18 This file describes the configuration scheme such a frontend can use to
  19 instruct the Acquire system to download those additional files.
  20
  21 # The Configuration Stanza
  22
  23 The Acquire system uses the same configuration settings to implement the
  24 files it downloads by default. These settings are the default, but if
  25 they would be written in a configuration file the configuration
  26 instructing the Acquire system to download the Packages files would look
  27 like this (see also apt.conf(5) manpage for configuration file syntax):
  28
  29         Acquire::IndexTargets::deb::Packages {
  30                 MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
  31                 ShortDescription "Packages";
  32                 Description "$(SITE) $(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";
  33
  34                 flatMetaKey "Packages";
  35                 flatDescription "$(SITE) $(RELEASE) Packages";
  36
  37                 Optional "false";
  38         };
  39
  40 All files which should be downloaded (nicknamed 'Targets') are mentioned
  41 below the Acquire::IndexTargets scope. 'deb' is here the type of the
  42 sources.list entry the file should be acquired for. The only other
  43 supported value is hence 'deb-src'. Beware: You can't specify multiple
  44 types here and you can't download the same (evaluated) MetaKey from
  45 multiple types!
  46
  47 After the type you can pick any valid and unique string which preferable
  48 refers to the file it downloads (In the example we picked 'Packages').
  49 This string is used as identifier for the target class and accessible as
  50 'Created-By' e.g. in the "apt-get indextargets" output as detailed below.
  51
  52 All targets have three main properties you can define:
  53 * MetaKey: The identifier of the file to be downloaded as used in the
  54   Release file.  It is also the relative location of the file from the
  55   Release file.  You can neither download from a different server
  56   entirely (absolute URI) nor access directories above the Release file
  57   (e.g. "../../").
  58 * ShortDescription: Very short string intended to be displayed to the
  59   user e.g.  while reporting progress. apt will e.g. use this string in
  60   the last line to indicate progress of e.g. the download of a specific
  61   item.
  62 * Description: A preferable human understandable and readable identifier
  63   of which file is acquired exactly. Mainly used for progress reporting
  64   and error messages. apt will e.g. use this string in the Get/Hit/Err
  65   progress lines.
  66
  67 Additional optional properties:
  68 * flat{MetaKey,Description}: APT supports two types of repositories:
  69   dists-style repositories which are the default and by far the most
  70   common which are named after the fact that the files are in an
  71   elaborated directory structure.  In contrast a flat-style repositories
  72   lumps all files together in one directory.  Support for these flat
  73   repositories exists mainly for legacy purposes only.  It is therefore
  74   recommend to not set these values.
  75 * Optional: The default value is 'true' and should be kept at this
  76   value.  If enabled the acquire system will skip the download if the
  77   file isn't mentioned in the Release file. Otherwise this is treated as
  78   a hard error and the update process fails. Note that failures while
  79   downloading (e.g. 404 or hash verification errors) are failures,
  80   regardless of this setting.
  81
  82
  83 The acquire system will automatically choose to download a compressed
  84 file if it is available and uncompress it for you, just as it will also
  85 use pdiff patching if provided by the repository and enabled by the
  86 user. You only have to ensure that the Release file contains the
  87 information about the compressed files/pdiffs to make this happen.
  88 NO properties have to be set to enable this.
  89
  90 # More examples
  91
  92 The stanzas for Translation-* files as well as for Sources files would
  93 look like this:
  94
  95 Acquire::IndexTargets {
  96         deb::Translations {
  97                 MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)";
  98                 ShortDescription "Translation-$(LANGUAGE)";
  99                 Description "$(SITE) $(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)";
 100
 101                 flatMetaKey "$(LANGUAGE)";
 102                 flatDescription "$(SITE) $(RELEASE) Translation-$(LANGUAGE)";
 103         };
 104
 105         deb-src::Sources {
 106                 MetaKey "$(COMPONENT)/source/Sources";
 107                 ShortDescription "Sources";
 108                 Description "$(SITE) $(RELEASE)/$(COMPONENT) Sources";
 109
 110                 flatMetaKey "Sources";
 111                 flatDescription "$(SITE) $(RELEASE) Sources";
 112
 113                 Optional "false";
 114         };
 115 };
 116
 117 # Substitution variables
 118
 119 As seen in the examples, properties can contain placeholders filled in
 120 by the acquire system. The following variables are known; note that
 121 unknown variables have no default value nor are they touched: They are
 122 printed as-is.
 123
 124 * $(SITE): An identifier of the site we access as seen in sources.list,
 125   e.g. "http://example.org/debian" or "file:/path/to/a/repository". You
 126   can't use this field in {,flat}MetaKey, it is for description proposes
 127   only.
 128 * $(RELEASE): This is usually an archive- or codename, e.g. "stable" or
 129   "stretch".  Note that flat-style repositories do not have a archive-
 130   or codename per-se, so the value might very well be just "/" or so.
 131   Again, as seen in the sources.list.
 132 * $(COMPONENT): as given in the sources.list, e.g. "main", "non-free" or
 133   "universe".  Note that flat-style repositories again do not really
 134   have a meaningful value here.
 135 * $(LANGUAGE): Values are all entries (expect "none") of configuration
 136   option Acquire::Languages, e.g. "en", "de" or "de_AT".
 137 * $(ARCHITECTURE): Values are all entries of configuration option
 138   APT::Architectures (potentially modified by sources.list options),
 139   e.g. "amd64", "i386" or "armel" for the 'deb' type. In type 'deb-src'
 140   this variable has the value "source".
 141
 142 Note that while more variables might exist in the implementation, these
 143 are to be considered undefined and their usage strongly discouraged. If
 144 you have a need for other variables contact us.
 145
 146 # Accessing files
 147
 148 Do NOT hardcode specific file locations, names or compression types in
 149 your application! You will notice that the configuration options give
 150 you no choice over where the downloaded files will be stored. This is by
 151 design so multiple applications can download and use the same file
 152 rather than each and every one of them potentially downloads and uses
 153 its own copy somewhere on disk.
 154
 155 "apt-get indextargets" can be used to get the location as well as other
 156 information about all files downloaded (aka: you will see Packages,
 157 Sources and Translation-* files here as well). Provide a line of the
 158 default output format as parameter to filter out all entries which do
 159 not have such a line. With --format, you can further more define your
 160 own output style. The variables are what you see in the output, just all
 161 uppercase and wrapped in $(), as in the configuration file.
 162
 163 To get all the filenames of all Translation-en files you can e.g. call:
 164         apt-get indextargets --format '$(FILENAME)' "Created-By: Translations" "Language: en"
 165
 166 The line-based filtering and the formating is rather crude and feature-
 167 less by design, so it is recommend to use dedicated and more powerful
 168 tools like 'grep-dctrl'.
 169
 170 Accessing this information via libapt is done by reading the
 171 sources.lists (pkgSourceList), iterating over the metaIndex objects this
 172 creates and calling GetIndexTargets() on them. See the sourcecode of
 173 "apt-get indextargets" for a complete example.
 174
 175 Note that by default targets are not listed if they weren't downloaded.
 176 If you want to see all targets, you can use the --no-release-info, which
 177 also removes the Codename, Suite, Version, Origin, Label and Trusted
 178 fields from the output as these also display data which needs to be
 179 downloaded first and could hence be inaccurate [on the pro-side: This
 180 mode is faster as it doesn't require a valid binary cache to operate].
 181 The most notable difference perhaps is in the Filename field through: By
 182 default it indicates an existing file, potentially compressed (Hint:
 183 libapt users can use FileFd to open compressed files transparently). In
 184 the --no-release-info mode the indicated file doesn't need to exist and
 185 it will always refer to an uncompressed file, even if the index would be
 186 (or is) stored compressed.
 187
 188 Remarks on fields only available in (default) --release-info mode:
 189 * Trusted: Denotes with a 'yes' or 'no' if the data in this file is
 190   authenticated by a trustchain rooted in a trusted gpg key. You should
 191   be careful with untrusted data and warn the user if you use it.
 192 * Codename, Suite, Version, Origin and Label are fields from the Release
 193   file, are only present if they are present in the Release file and
 194   contain the same data.
 195
 196 Remarks on other available fields:
 197 * MetaKey, ShortDesc, Description, Site, Release: as defined
 198   by the configuration and described further above.
 199 * Created-By: configuration entity responsible for this target
 200 * Target-Of: type of the sources.list entry
 201 * URI, Repo-URI: avoid using. Contains potentially username/password.
 202   Prefer 'Site', especially for display.
 203 * Optional: Decodes the option of the same name from the configuration.
 204   Note that it is using 'yes' and 'no' instead of 'true' and 'false'.
 205 * Language, Architecture, Component: as defined further above, but with
 206   the catch that they might be missing if they don't effect the target
 207   (aka: They weren't used while evaluating the MetaKey template).
 208
 209 Again, additional fields might be visible in certain implementations,
 210 but you should avoid using them and instead talk to us about a portable
 211 implementation.
 212
 213 # Multiple application requiring the same files
 214
 215 It is highly encouraged that applications talk to each other and to us
 216 about which files they require. It is usually best to have a common
 217 package ship the configuration needed to get the files, but specific
 218 needs might require specific solutions. Again: talk to us.
 219
 220 # Acquiring files not mentioned in the Release file
 221
 222 You can't. This is by design as these files couldn't be verified to not
 223 be modified in transit, corrupted by the download process or simple if
 224 they are present at all on the server, which would require apt to probe
 225 for them. APT did this in the past for legacy reasons, we do not intend
 226 to go back to these dark times.
 227
 228 This is also why you can't request files from a different server. It
 229 would have the additional problem that this server might not even be
 230 accessible (e.g. proxy settings) or that local sources (file:/, cdrom:/)
 231 start requesting online files…
 232
 233 In other words: We would be opening Pandora's box.