doc/acquire-additional-files.txt

   1 # Acquire additional files in 'update' operations
   2
   3 The download and verification of data from multiple sources in different
   4 compression formats, with partial downloads and patches is an involved
   5 process which is hard to implement correctly and securely.
   6
   7 APT frontends share the code and binaries to make this happen in libapt
   8 with the Acquire system, supported by helpers shipped in the apt package
   9 itself and additional transports in individual packages like
  10 apt-transport-https.
  11
  12 For its own operation libapt needs or can make use of Packages, Sources
  13 and Translation-* files, which it will acquire by default, but
  14 a repository might contain more data files (e.g.  Contents) a frontend
  15 might want to use and would therefore need to be downloaded as well
  16 (e.g. apt-file).
  17
  18 This file describes the configuration scheme such a frontend can use to
  19 instruct the Acquire system to download those additional files.
  20
  21 # The Configuration Stanza
  22
  23 The Acquire system uses the same configuration settings to implement the
  24 files it downloads by default. These settings are the default, but if
  25 they would be written in a configuration file the configuration
  26 instructing the Acquire system to download the Packages files would look
  27 like this (see also apt.conf(5) manpage for configuration file syntax):
  28
  29         Acquire::IndexTargets::deb::Packages {
  30                 MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
  31                 ShortDescription "Packages";
  32                 Description "$(SITE) $(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";
  33
  34                 flatMetaKey "Packages";
  35                 flatDescription "$(SITE) $(RELEASE) Packages";
  36
  37                 Optional "false";
  38         };
  39
  40 All files which should be downloaded (nicknamed 'Targets') are mentioned
  41 below the Acquire::IndexTargets scope. 'deb' is here the type of the
  42 sources.list entry the file should be acquired for. The only other
  43 supported value is hence 'deb-src'. Beware: You can't specify multiple
  44 types here and you can't download the same (evaluated) MetaKey from
  45 multiple types!
  46
  47 After the type you can pick any valid and unique string which preferable
  48 refers to the file it downloads (In the example we picked 'Packages').
  49 This string is used as identifier for the target class and accessible as
  50 'Created-By' e.g. in the "apt-get indextargets" output as detailed below.
  51
  52 All targets have three main properties you can define:
  53 * MetaKey: The identifier of the file to be downloaded as used in the
  54   Release file.  It is also the relative location of the file from the
  55   Release file.  You can neither download from a different server
  56   entirely (absolute URI) nor access directories above the Release file
  57   (e.g. "../../").
  58 * ShortDescription: Very short string intended to be displayed to the
  59   user e.g.  while reporting progress. apt will e.g. use this string in
  60   the last line to indicate progress of e.g. the download of a specific
  61   item.
  62 * Description: A preferable human understandable and readable identifier
  63   of which file is acquired exactly. Mainly used for progress reporting
  64   and error messages. apt will e.g. use this string in the Get/Hit/Err
  65   progress lines.
  66
  67 Additional optional properties:
  68 * flat{MetaKey,Description}: APT supports two types of repositories:
  69   dists-style repositories which are the default and by far the most
  70   common which are named after the fact that the files are in an
  71   elaborated directory structure.  In contrast a flat-style repositories
  72   lumps all files together in one directory.  Support for these flat
  73   repositories exists mainly for legacy purposes only.  It is therefore
  74   recommend to not set these values.
  75 * Optional: The default value is 'true' and should be kept at this
  76   value.  If enabled the acquire system will skip the download if the
  77   file isn't mentioned in the Release file. Otherwise this is treated as
  78   a hard error and the update process fails. Note that failures while
  79   downloading (e.g. 404 or hash verification errors) are failures,
  80   regardless of this setting.
  81
  82
  83 The acquire system will automatically choose to download a compressed
  84 file if it is available and uncompress it for you, just as it will also
  85 use pdiff patching if provided by the repository and enabled by the
  86 user. You only have to ensure that the Release file contains the
  87 information about the compressed files/pdiffs to make this happen.
  88 *NO* properties have to be set to enable this!
  89
  90
  91 Additional properties exist, but these should *NOT* be set by frontends
  92 requesting files. They exist for internal and end-user usage only:
  93 * PDiffs: controls if apt will try to use pdiffs for this target.
  94   Defaults to the value of Acquire::PDiffs which is true by default.
  95   Can be overridden per-source by the sources.list option of the same
  96   name. See the documentation for both of these for details.
  97
  98 # More examples
  99
 100 The stanzas for Translation-* files as well as for Sources files would
 101 look like this:
 102
 103 Acquire::IndexTargets {
 104         deb::Translations {
 105                 MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)";
 106                 ShortDescription "Translation-$(LANGUAGE)";
 107                 Description "$(SITE) $(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)";
 108
 109                 flatMetaKey "$(LANGUAGE)";
 110                 flatDescription "$(SITE) $(RELEASE) Translation-$(LANGUAGE)";
 111         };
 112
 113         deb-src::Sources {
 114                 MetaKey "$(COMPONENT)/source/Sources";
 115                 ShortDescription "Sources";
 116                 Description "$(SITE) $(RELEASE)/$(COMPONENT) Sources";
 117
 118                 flatMetaKey "Sources";
 119                 flatDescription "$(SITE) $(RELEASE) Sources";
 120
 121                 Optional "false";
 122         };
 123 };
 124
 125 # Substitution variables
 126
 127 As seen in the examples, properties can contain placeholders filled in
 128 by the acquire system. The following variables are known; note that
 129 unknown variables have no default value nor are they touched: They are
 130 printed as-is.
 131
 132 * $(SITE): An identifier of the site we access as seen in sources.list,
 133   e.g. "http://example.org/debian" or "file:/path/to/a/repository". You
 134   can't use this field in {,flat}MetaKey, it is for description proposes
 135   only.
 136 * $(RELEASE): This is usually an archive- or codename, e.g. "stable" or
 137   "stretch".  Note that flat-style repositories do not have a archive-
 138   or codename per-se, so the value might very well be just "/" or so.
 139   Again, as seen in the sources.list.
 140 * $(COMPONENT): as given in the sources.list, e.g. "main", "non-free" or
 141   "universe".  Note that flat-style repositories again do not really
 142   have a meaningful value here.
 143 * $(LANGUAGE): Values are all entries (expect "none") of configuration
 144   option Acquire::Languages, e.g. "en", "de" or "de_AT".
 145 * $(ARCHITECTURE): Values are all entries of configuration option
 146   APT::Architectures (potentially modified by sources.list options),
 147   e.g. "amd64", "i386" or "armel" for the 'deb' type. In type 'deb-src'
 148   this variable has the value "source".
 149
 150 Note that while more variables might exist in the implementation, these
 151 are to be considered undefined and their usage strongly discouraged. If
 152 you have a need for other variables contact us.
 153
 154 # Accessing files
 155
 156 Do NOT hardcode specific file locations, names or compression types in
 157 your application! You will notice that the configuration options give
 158 you no choice over where the downloaded files will be stored. This is by
 159 design so multiple applications can download and use the same file
 160 rather than each and every one of them potentially downloads and uses
 161 its own copy somewhere on disk.
 162
 163 "apt-get indextargets" can be used to get the location as well as other
 164 information about all files downloaded (aka: you will see Packages,
 165 Sources and Translation-* files here as well). Provide a line of the
 166 default output format as parameter to filter out all entries which do
 167 not have such a line. With --format, you can further more define your
 168 own output style. The variables are what you see in the output, just all
 169 uppercase and wrapped in $(), as in the configuration file.
 170
 171 To get all the filenames of all Translation-en files you can e.g. call:
 172         apt-get indextargets --format '$(FILENAME)' "Created-By: Translations" "Language: en"
 173
 174 The line-based filtering and the formating is rather crude and feature-
 175 less by design, so it is recommend to use dedicated and more powerful
 176 tools like 'grep-dctrl'.
 177
 178 Accessing this information via libapt is done by reading the
 179 sources.lists (pkgSourceList), iterating over the metaIndex objects this
 180 creates and calling GetIndexTargets() on them. See the sourcecode of
 181 "apt-get indextargets" for a complete example.
 182
 183 Note that by default targets are not listed if they weren't downloaded.
 184 If you want to see all targets, you can use the --no-release-info, which
 185 also removes the Codename, Suite, Version, Origin, Label and Trusted
 186 fields from the output as these also display data which needs to be
 187 downloaded first and could hence be inaccurate [on the pro-side: This
 188 mode is faster as it doesn't require a valid binary cache to operate].
 189 The most notable difference perhaps is in the Filename field through: By
 190 default it indicates an existing file, potentially compressed (Hint:
 191 libapt users can use FileFd to open compressed files transparently). In
 192 the --no-release-info mode the indicated file doesn't need to exist and
 193 it will always refer to an uncompressed file, even if the index would be
 194 (or is) stored compressed.
 195
 196 Remarks on fields only available in (default) --release-info mode:
 197 * Trusted: Denotes with a 'yes' or 'no' if the data in this file is
 198   authenticated by a trustchain rooted in a trusted gpg key. You should
 199   be careful with untrusted data and warn the user if you use it.
 200 * Codename, Suite, Version, Origin and Label are fields from the Release
 201   file, are only present if they are present in the Release file and
 202   contain the same data.
 203
 204 Remarks on other available fields:
 205 * MetaKey, ShortDesc, Description, Site, Release: as defined
 206   by the configuration and described further above.
 207 * Created-By: configuration entity responsible for this target
 208 * Target-Of: type of the sources.list entry
 209 * URI, Repo-URI: avoid using. Contains potentially username/password.
 210   Prefer 'Site', especially for display.
 211 * Optional: Decodes the option of the same name from the configuration.
 212   Note that it is using 'yes' and 'no' instead of 'true' and 'false'.
 213 * Language, Architecture, Component: as defined further above, but with
 214   the catch that they might be missing if they don't effect the target
 215   (aka: They weren't used while evaluating the MetaKey template).
 216
 217 Again, additional fields might be visible in certain implementations,
 218 but you should avoid using them and instead talk to us about a portable
 219 implementation.
 220
 221 # Multiple application requiring the same files
 222
 223 It is highly encouraged that applications talk to each other and to us
 224 about which files they require. It is usually best to have a common
 225 package ship the configuration needed to get the files, but specific
 226 needs might require specific solutions. Again: talk to us.
 227
 228 # Acquiring files not mentioned in the Release file
 229
 230 You can't. This is by design as these files couldn't be verified to not
 231 be modified in transit, corrupted by the download process or simple if
 232 they are present at all on the server, which would require apt to probe
 233 for them. APT did this in the past for legacy reasons, we do not intend
 234 to go back to these dark times.
 235
 236 This is also why you can't request files from a different server. It
 237 would have the additional problem that this server might not even be
 238 accessible (e.g. proxy settings) or that local sources (file:/, cdrom:/)
 239 start requesting online files…
 240
 241 In other words: We would be opening Pandora's box.