delay progress until Release files are downloaded

[apt.git] / doc / acquire-additional-files.txt

diff --git a/doc/acquire-additional-files.txt b/doc/acquire-additional-files.txt
index 71ce7b0cb917d5c23e76b40e208c0716ccb8195a..68af9a5b03c2dad6f376453b37ac61e296ec49cd 100644 (file)
--- a/doc/acquire-additional-files.txt
+++ b/doc/acquire-additional-files.txt
@@ -4,18 +4,18 @@ The download and verification of data from multiple sources in different
 compression formats, with partial downloads and patches is an involved
 process which is hard to implement correctly and securely.
 
 compression formats, with partial downloads and patches is an involved
 process which is hard to implement correctly and securely.
 

-APT frontends share the code and binaries to make this happen in libapt
+APT front-ends share the code and binaries to make this happen in libapt

 with the Acquire system, supported by helpers shipped in the apt package
 itself and additional transports in individual packages like
 apt-transport-https.
 
 For its own operation libapt needs or can make use of Packages, Sources
 and Translation-* files, which it will acquire by default, but
 with the Acquire system, supported by helpers shipped in the apt package
 itself and additional transports in individual packages like
 apt-transport-https.
 
 For its own operation libapt needs or can make use of Packages, Sources
 and Translation-* files, which it will acquire by default, but

-a repository might contain more data files (e.g.  Contents) a frontend
-might want to use and would therefore need to be downloaded as well
-(e.g. apt-file).
+a repository might contain more data files (e.g.  Contents) a front-end
+(e.g. apt-file) might want to use and would therefore need to be
+downloaded as well.

 
 

-This file describes the configuration scheme such a frontend can use to
+This file describes the configuration scheme such a front-end can use to

 instruct the Acquire system to download those additional files.
 
 # The Configuration Stanza
 instruct the Acquire system to download those additional files.
 
 # The Configuration Stanza


@@ -29,12 +29,12 @@ like this (see also apt.conf(5) manpage for configuration file syntax):
        Acquire::IndexTargets::deb::Packages {
                MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
                ShortDescription "Packages";
        Acquire::IndexTargets::deb::Packages {
                MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
                ShortDescription "Packages";

-               Description "$(SITE) $(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";
+               Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";

 
                flatMetaKey "Packages";
 
                flatMetaKey "Packages";

-               flatDescription "$(SITE) $(RELEASE) Packages";
+               flatDescription "$(RELEASE) Packages";

 
 

-               Optional "false";
+               Optional "no";

        };
 
 All files which should be downloaded (nicknamed 'Targets') are mentioned
        };
 
 All files which should be downloaded (nicknamed 'Targets') are mentioned


@@ -47,7 +47,9 @@ multiple types!
 After the type you can pick any valid and unique string which preferable
 refers to the file it downloads (In the example we picked 'Packages').
 This string is used as identifier for the target class and accessible as
 After the type you can pick any valid and unique string which preferable
 refers to the file it downloads (In the example we picked 'Packages').
 This string is used as identifier for the target class and accessible as

-'Created-By' e.g. in the "apt-get indextargets" output as detailed below.
+'Created-By' e.g. in the "apt-get indextargets" output as detailed
+below.  It is also used to allow user to enable/disable targets per
+sources.list entry.

 
 All targets have three main properties you can define:
 * MetaKey: The identifier of the file to be downloaded as used in the
 
 All targets have three main properties you can define:
 * MetaKey: The identifier of the file to be downloaded as used in the


@@ -63,29 +65,70 @@ All targets have three main properties you can define:
   of which file is acquired exactly. Mainly used for progress reporting
   and error messages. apt will e.g. use this string in the Get/Hit/Err
   progress lines.
   of which file is acquired exactly. Mainly used for progress reporting
   and error messages. apt will e.g. use this string in the Get/Hit/Err
   progress lines.

+  An identifier of the site accessed as seen in the sources.list (e.g.
+  "http://example.org/debian" or "file:/path/to/a/repository") is
+  automatically prefixed for this property.
+

 
 Additional optional properties:
 
 Additional optional properties:

+* DefaultEnabled: The default value is 'yes' which means that apt will
+  try to acquire this target from all sources. If set to 'no' the user
+  has to explicitly enable this target in the sources.list file with the
+  Targets option(s) – or override this value in a config file.
+* Optional: The default value is 'yes' and should be kept at this value.
+  If enabled the acquire system will skip the download if the file isn't
+  mentioned in the Release file. Otherwise this is treated as a hard
+  error and the update process fails. Note that failures while
+  downloading (e.g. 404 or hash verification errors) are failures,
+  regardless of this setting.
+* KeepCompressed: The default is the value of Acquire::GzipIndexes,
+  which defaults to false. If true, the acquire system will keep the
+  file compressed on disk rather than extract it. If your front-end can't
+  deal with compressed files transparently you have to explicitly set
+  this option to false to avoid problems with users setting the option
+  globally. On the other hand, if you set it to true or don't set it you
+  have to ensure your front-end can deal with all compressed fileformats
+  supported by apt (libapt users can e.g. use FileFd, others can use
+  the cat-file command of /usr/lib/apt/apt-helper).

 * flat{MetaKey,Description}: APT supports two types of repositories:
   dists-style repositories which are the default and by far the most
   common which are named after the fact that the files are in an
 * flat{MetaKey,Description}: APT supports two types of repositories:
   dists-style repositories which are the default and by far the most
   common which are named after the fact that the files are in an

-  elaborated directory structure.  In contrast a flat-style repositories
+  elaborated directory structure.  In contrast a flat-style repository

   lumps all files together in one directory.  Support for these flat
   repositories exists mainly for legacy purposes only.  It is therefore
   recommend to not set these values.
   lumps all files together in one directory.  Support for these flat
   repositories exists mainly for legacy purposes only.  It is therefore
   recommend to not set these values.

-* Optional: The default value is 'true' and should be kept at this
-  value.  If enabled the acquire system will skip the download if the
-  file isn't mentioned in the Release file. Otherwise this is treated as
-  a hard error and the update process fails. Note that failures while
-  downloading (e.g. 404 or hash verification errors) are failures,
-  regardless of this setting.

 
 
 The acquire system will automatically choose to download a compressed
 file if it is available and uncompress it for you, just as it will also
 
 
 The acquire system will automatically choose to download a compressed
 file if it is available and uncompress it for you, just as it will also

-use pdiff patching if provided by the repository and enabled by the
+use PDiff patching if provided by the repository and enabled by the

 user. You only have to ensure that the Release file contains the
 user. You only have to ensure that the Release file contains the

-information about the compressed files/pdiffs to make this happen.
-NO properties have to be set to enable this.
+information about the compressed files/PDiffs to make this happen.
+*NO* properties have to be set to enable this!
+
+
+More properties exist, but these should *NOT* be set by front-ends
+requesting files. They exist for internal and end-user usage only.
+Some of these are – which are documented here only to ensure that they
+aren't accidentally used by front-ends:
+* PDiffs: controls if apt will try to use PDiffs for this target.
+  Defaults to the value of Acquire::PDiffs which is true by default.
+  Can be overridden per-source by the sources.list option of the same
+  name. See the documentation for both of these for details.
+* By-Hash: controls if apt will try to use an URI constructed from
+  a hashsum of the file to download. See the documentation for config
+  option Acquire::By-Hash and sources.list option By-Hash for details.
+* CompressionTypes: The default value is a space separated list of
+  compression types supported by apt (see Acquire::CompressionTypes).
+  You can set this option to prevent apt from downloading a compression
+  type a front-end can't open transparently. This should always be
+  a temporary workaround through and a bug should be reported against
+  the front-end in question.
+* KeepCompressedAs: The default value is a space separated list of
+  compression types supported by apt (see previous option) which is
+  sorted by the cost-value of the compression in ascending order,
+  except that cost=0 "compressions" (like uncompressed) are listed last.
+

 
 # More examples
 
 
 # More examples
 


@@ -96,21 +139,21 @@ Acquire::IndexTargets {
        deb::Translations {
                MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)";
                ShortDescription "Translation-$(LANGUAGE)";
        deb::Translations {
                MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)";
                ShortDescription "Translation-$(LANGUAGE)";

-               Description "$(SITE) $(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)";
+               Description "$(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)";

 
                flatMetaKey "$(LANGUAGE)";
 
                flatMetaKey "$(LANGUAGE)";

-               flatDescription "$(SITE) $(RELEASE) Translation-$(LANGUAGE)";
+               flatDescription "$(RELEASE) Translation-$(LANGUAGE)";

        };
 
        deb-src::Sources {
                MetaKey "$(COMPONENT)/source/Sources";
                ShortDescription "Sources";
        };
 
        deb-src::Sources {
                MetaKey "$(COMPONENT)/source/Sources";
                ShortDescription "Sources";

-               Description "$(SITE) $(RELEASE)/$(COMPONENT) Sources";
+               Description "$(RELEASE)/$(COMPONENT) Sources";

 
                flatMetaKey "Sources";
 
                flatMetaKey "Sources";

-               flatDescription "$(SITE) $(RELEASE) Sources";
+               flatDescription "$(RELEASE) Sources";

 
 

-               Optional "false";
+               Optional "no";

        };
 };
 
        };
 };
 


@@ -121,14 +164,9 @@ by the acquire system. The following variables are known; note that
 unknown variables have no default value nor are they touched: They are
 printed as-is.
 
 unknown variables have no default value nor are they touched: They are
 printed as-is.
 

-* $(SITE): An identifier of the site we access as seen in sources.list,
-  e.g. "http://example.org/debian" or "file:/path/to/a/repository". You
-  can't use this field in {,flat}MetaKey, it is for description proposes
-  only.

 * $(RELEASE): This is usually an archive- or codename, e.g. "stable" or
 * $(RELEASE): This is usually an archive- or codename, e.g. "stable" or

-  "stretch".  Note that flat-style repositories do not have a archive-
+  "stretch".  Note that flat-style repositories do not have an archive-

   or codename per-se, so the value might very well be just "/" or so.
   or codename per-se, so the value might very well be just "/" or so.

-  Again, as seen in the sources.list.

 * $(COMPONENT): as given in the sources.list, e.g. "main", "non-free" or
   "universe".  Note that flat-style repositories again do not really
   have a meaningful value here.
 * $(COMPONENT): as given in the sources.list, e.g. "main", "non-free" or
   "universe".  Note that flat-style repositories again do not really
   have a meaningful value here.


@@ -138,6 +176,9 @@ printed as-is.
   APT::Architectures (potentially modified by sources.list options),
   e.g. "amd64", "i386" or "armel" for the 'deb' type. In type 'deb-src'
   this variable has the value "source".
   APT::Architectures (potentially modified by sources.list options),
   e.g. "amd64", "i386" or "armel" for the 'deb' type. In type 'deb-src'
   this variable has the value "source".

+* $(NATIVE_ARCHITECTURE): The architecture apt treats as the native
+  architecture for this system configured as APT::Architecture
+  defaulting to the architecture apt itself was built for.

 
 Note that while more variables might exist in the implementation, these
 are to be considered undefined and their usage strongly discouraged. If
 
 Note that while more variables might exist in the implementation, these
 are to be considered undefined and their usage strongly discouraged. If


@@ -164,12 +205,16 @@ To get all the filenames of all Translation-en files you can e.g. call:
        apt-get indextargets --format '$(FILENAME)' "Created-By: Translations" "Language: en"
 
 The line-based filtering and the formating is rather crude and feature-
        apt-get indextargets --format '$(FILENAME)' "Created-By: Translations" "Language: en"
 
 The line-based filtering and the formating is rather crude and feature-

-less by design, so it is recommend to use dedicated and more powerful
-tools like 'grep-dctrl'.
+less by design: The default format is Debians standard format deb822 (in
+particular: Field names are case-insensitive and the order of fields in
+the stanza is undefined), so instead of apt reimplementing powerful
+filters and formating for this command, it is recommend to use piping
+and dedicated tools like 'grep-dctrl' if you need more than the basics
+provided.

 
 Accessing this information via libapt is done by reading the
 sources.lists (pkgSourceList), iterating over the metaIndex objects this
 
 Accessing this information via libapt is done by reading the
 sources.lists (pkgSourceList), iterating over the metaIndex objects this

-creates and calling GetIndexTargets() on them. See the sourcecode of
+creates and calling GetIndexTargets() on them. See the source code of

 "apt-get indextargets" for a complete example.
 
 Note that by default targets are not listed if they weren't downloaded.
 "apt-get indextargets" for a complete example.
 
 Note that by default targets are not listed if they weren't downloaded.


@@ -187,7 +232,7 @@ it will always refer to an uncompressed file, even if the index would be
 
 Remarks on fields only available in (default) --release-info mode:
 * Trusted: Denotes with a 'yes' or 'no' if the data in this file is
 
 Remarks on fields only available in (default) --release-info mode:
 * Trusted: Denotes with a 'yes' or 'no' if the data in this file is

-  authenticated by a trustchain rooted in a trusted gpg key. You should
+  authenticated by a trust chain rooted in a trusted gpg key. You should

   be careful with untrusted data and warn the user if you use it.
 * Codename, Suite, Version, Origin and Label are fields from the Release
   file, are only present if they are present in the Release file and
   be careful with untrusted data and warn the user if you use it.
 * Codename, Suite, Version, Origin and Label are fields from the Release
   file, are only present if they are present in the Release file and


@@ -200,8 +245,8 @@ Remarks on other available fields:
 * Target-Of: type of the sources.list entry
 * URI, Repo-URI: avoid using. Contains potentially username/password.
   Prefer 'Site', especially for display.
 * Target-Of: type of the sources.list entry
 * URI, Repo-URI: avoid using. Contains potentially username/password.
   Prefer 'Site', especially for display.

-* Optional: Decodes the option of the same name from the configuration.
-  Note that it is using 'yes' and 'no' instead of 'true' and 'false'.
+* Optional, DefaultEnabled, KeepCompressed: Decode the options of the
+  same name from the configuration.

 * Language, Architecture, Component: as defined further above, but with
   the catch that they might be missing if they don't effect the target
   (aka: They weren't used while evaluating the MetaKey template).
 * Language, Architecture, Component: as defined further above, but with
   the catch that they might be missing if they don't effect the target
   (aka: They weren't used while evaluating the MetaKey template).


@@ -210,13 +255,17 @@ Again, additional fields might be visible in certain implementations,
 but you should avoid using them and instead talk to us about a portable
 implementation.
 
 but you should avoid using them and instead talk to us about a portable
 implementation.
 

-# Multiple application requiring the same files
+# Multiple applications requiring the same files

 
 It is highly encouraged that applications talk to each other and to us
 about which files they require. It is usually best to have a common
 package ship the configuration needed to get the files, but specific
 needs might require specific solutions. Again: talk to us.
 
 
 It is highly encouraged that applications talk to each other and to us
 about which files they require. It is usually best to have a common
 package ship the configuration needed to get the files, but specific
 needs might require specific solutions. Again: talk to us.
 

+Bad things will happen if multiple front-ends request the same file(s)
+via different targets, which is another reason why coordination is very
+important!
+

 # Acquiring files not mentioned in the Release file
 
 You can't. This is by design as these files couldn't be verified to not
 # Acquiring files not mentioned in the Release file
 
 You can't. This is by design as these files couldn't be verified to not


@@ -231,3 +280,25 @@ accessible (e.g. proxy settings) or that local sources (file:/, cdrom:/)
 start requesting online files…
 
 In other words: We would be opening Pandora's box.
 start requesting online files…
 
 In other words: We would be opening Pandora's box.

+
+# Acquiring files to a specific location on disk
+
+You can't by design to avoid multiple front-ends requesting the same file
+to be downloaded to multiple different places on (different) disks
+(among other reasons).  See the next point for a solution if you really
+have to force a specific location by creating symlinks.
+
+# Post processing the acquired files
+
+You can't modify the files apt has downloaded as apt keeps state with
+e.g. the modification times of the files and advanced features like
+PDiffs break.
+
+You can however install an APT::Update::Post-Invoke{-Success,} hook
+script and use them to copy (modified) files to a different location.
+Use 'apt-get indextargets' (or similar) to get the filenames – do not
+look into /var/lib/apt/lists directly!
+
+Please avoid time consuming calculations in the scripts and instead just
+trigger a background task as there is little to no feedback for the user
+while hook scripts run.