doc/apt-ftparchive.1.xml

   1 <?xml version="1.0" encoding="utf-8" standalone="no"?>
   2 <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
   3   "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [
   4
   5 <!ENTITY % aptent SYSTEM "apt.ent">
   6 %aptent;
   7
   8 ]>
   9
  10 <refentry>
  11
  12  <refentryinfo>
  13    &apt-author.jgunthorpe;
  14    &apt-author.team;
  15    &apt-email;
  16    &apt-product;
  17    <!-- The last update date -->
  18    <date>17 August 2009</date>
  19  </refentryinfo>
  20
  21  <refmeta>
  22    <refentrytitle>apt-ftparchive</refentrytitle>
  23    <manvolnum>1</manvolnum>
  24    <refmiscinfo class="manual">APT</refmiscinfo>
  25  </refmeta>
  26
  27  <!-- Man page title -->
  28  <refnamediv>
  29     <refname>apt-ftparchive</refname>
  30     <refpurpose>Utility to generate index files</refpurpose>
  31  </refnamediv>
  32
  33  <!-- Arguments -->
  34  <refsynopsisdiv>
  35    <cmdsynopsis>
  36       <command>apt-ftparchive</command>
  37       <arg><option>-hvdsq</option></arg>
  38       <arg><option>--md5</option></arg>
  39       <arg><option>--delink</option></arg>
  40       <arg><option>--readonly</option></arg>
  41       <arg><option>--contents</option></arg>
  42       <arg><option>-o <replaceable>config</replaceable>=<replaceable>string</replaceable></option></arg>
  43       <arg><option>-c=<replaceable>file</replaceable></option></arg>
  44       <group choice="req">
  45          <arg>packages<arg choice="plain" rep="repeat"><replaceable>path</replaceable></arg><arg><replaceable>override</replaceable><arg><replaceable>pathprefix</replaceable></arg></arg></arg>
  46          <arg>sources<arg choice="plain" rep="repeat"><replaceable>path</replaceable></arg><arg><replaceable>override</replaceable><arg><replaceable>pathprefix</replaceable></arg></arg></arg>
  47          <arg>contents <arg choice="plain"><replaceable>path</replaceable></arg></arg>
  48          <arg>release <arg choice="plain"><replaceable>path</replaceable></arg></arg>
  49          <arg>generate <arg choice="plain"><replaceable>config-file</replaceable></arg> <arg choice="plain" rep="repeat"><replaceable>section</replaceable></arg></arg>
  50          <arg>clean <arg choice="plain"><replaceable>config-file</replaceable></arg></arg>
  51       </group>
  52    </cmdsynopsis>
  53  </refsynopsisdiv>
  54
  55  <refsect1><title>Description</title>
  56    <para><command>apt-ftparchive</command> is the command line tool that generates the index
  57    files that APT uses to access a distribution source. The index files should
  58    be generated on the origin site based on the content of that site.</para>
  59
  60    <para><command>apt-ftparchive</command> is a superset of the &dpkg-scanpackages; program,
  61    incorporating its entire functionality via the <literal>packages</literal> command.
  62    It also contains a contents file generator, <literal>contents</literal>, and an
  63    elaborate means to 'script' the generation process for a complete
  64    archive.</para>
  65
  66    <para>Internally <command>apt-ftparchive</command> can make use of binary databases to
  67    cache the contents of a .deb file and it does not rely on any external
  68    programs aside from &gzip;. When doing a full generate it automatically
  69    performs file-change checks and builds the desired compressed output files.</para>
  70
  71    <para>Unless the <option>-h</option>, or <option>--help</option> option is given one of the
  72    commands below must be present.</para>
  73
  74    <variablelist>
  75      <varlistentry><term>packages</term>
  76      <listitem><para>
  77      The packages command generates a package file from a directory tree. It
  78      takes the given directory and recursively searches it for .deb files,
  79      emitting a package record to stdout for each. This command is
  80      approximately equivalent to &dpkg-scanpackages;.</para>
  81
  82      <para>The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
  83      </varlistentry>
  84
  85      <varlistentry><term>sources</term>
  86      <listitem><para>
  87      The <literal>sources</literal> command generates a source index file from a directory tree.
  88      It takes the given directory and recursively searches it for .dsc files,
  89      emitting a source record to stdout for each. This command is approximately
  90      equivalent to &dpkg-scansources;.</para>
  91      <para>
  92      If an override file is specified then a source override file will be
  93      looked for with an extension of .src. The --source-override option can be
  94      used to change the source override file that will be used.</para></listitem>
  95      </varlistentry>
  96
  97      <varlistentry><term>contents</term>
  98      <listitem><para>
  99      The <literal>contents</literal> command generates a contents file from a directory tree. It
 100      takes the given directory and recursively searches it for .deb files,
 101      and reads the file list from each file. It then sorts and writes to stdout
 102      the list of files matched to packages. Directories are not written to
 103      the output. If multiple packages own the same file then each package is
 104      separated by a comma in the output.</para>
 105      <para>
 106      The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
 107      </varlistentry>
 108
 109      <varlistentry><term>release</term>
 110      <listitem><para>
 111      The <literal>release</literal> command generates a Release file from a
 112      directory tree. It recursively searches the given directory for
 113      Packages, Packages.gz, Packages.bz2, Sources, Sources.gz,
 114      Sources.bz2, Release and md5sum.txt files.  It then writes to
 115      stdout a Release file containing an MD5 digest and SHA1 digest
 116      for each file.</para>
 117      <para>
 118      Values for the additional metadata fields in the Release file are
 119      taken from the corresponding variables under
 120      <literal>APT::FTPArchive::Release</literal>,
 121      e.g. <literal>APT::FTPArchive::Release::Origin</literal>.  The supported fields
 122      are: <literal>Origin</literal>, <literal>Label</literal>, <literal>Suite</literal>,
 123      <literal>Version</literal>, <literal>Codename</literal>, <literal>Date</literal>,
 124      <literal>Architectures</literal>, <literal>Components</literal>, <literal>Description</literal>.</para></listitem>
 125
 126      </varlistentry>
 127
 128      <varlistentry><term>generate</term>
 129      <listitem><para>
 130      The <literal>generate</literal> command is designed to be runnable from a cron script and
 131      builds indexes according to the given config file. The config language
 132      provides a flexible means of specifying which index files are built from
 133      which directories, as well as providing a simple means of maintaining the
 134      required settings.</para></listitem>
 135      </varlistentry>
 136
 137      <varlistentry><term>clean</term>
 138      <listitem><para>
 139      The <literal>clean</literal> command tidies the databases used by the given
 140      configuration file by removing any records that are no longer necessary.</para></listitem>
 141      </varlistentry>
 142    </variablelist>
 143  </refsect1>
 144
 145  <refsect1><title>The Generate Configuration</title>
 146    <para>
 147    The <literal>generate</literal> command uses a configuration file to describe the
 148    archives that are going to be generated. It follows the typical ISC
 149    configuration format as seen in ISC tools like bind 8 and dhcpd.
 150    &apt-conf; contains a description of the syntax. Note that the generate
 151    configuration is parsed in sectional manner, but &apt-conf; is parsed in a
 152    tree manner. This only effects how the scope tag is handled.</para>
 153
 154    <para>
 155    The generate configuration has 4 separate sections, each described below.</para>
 156
 157    <refsect2><title>Dir Section</title>
 158      <para>
 159      The <literal>Dir</literal> section defines the standard directories needed to
 160      locate the files required during the generation process. These
 161      directories are prepended certain relative paths defined in later
 162      sections to produce a complete an absolute path.</para>
 163      <variablelist>
 164       <varlistentry><term>ArchiveDir</term>
 165       <listitem><para>
 166       Specifies the root of the FTP archive, in a standard
 167       Debian configuration this is the directory that contains the
 168       <filename>ls-LR</filename> and dist nodes.</para></listitem>
 169       </varlistentry>
 170
 171       <varlistentry><term>OverrideDir</term>
 172       <listitem><para>
 173       Specifies the location of the override files.</para></listitem>
 174       </varlistentry>
 175
 176       <varlistentry><term>CacheDir</term>
 177       <listitem><para>
 178       Specifies the location of the cache files</para></listitem>
 179       </varlistentry>
 180
 181       <varlistentry><term>FileListDir</term>
 182       <listitem><para>
 183       Specifies the location of the file list files,
 184       if the <literal>FileList</literal> setting is used below.</para></listitem>
 185       </varlistentry>
 186      </variablelist>
 187    </refsect2>
 188
 189    <refsect2><title>Default Section</title>
 190      <para>
 191      The <literal>Default</literal> section specifies default values, and settings
 192      that control the operation of the generator. Other sections may override
 193      these defaults with a per-section setting.</para>
 194      <variablelist>
 195       <varlistentry><term>Packages::Compress</term>
 196       <listitem><para>
 197       Sets the default compression schemes to use
 198       for the Package index files. It is a string that contains a space
 199       separated list of at least one of: '.' (no compression), 'gzip' and
 200       'bzip2'. The default for all compression schemes is '. gzip'.</para></listitem>
 201       </varlistentry>
 202
 203       <varlistentry><term>Packages::Extensions</term>
 204       <listitem><para>
 205       Sets the default list of file extensions that are package files.
 206       This defaults to '.deb'.</para></listitem>
 207       </varlistentry>
 208
 209       <varlistentry><term>Sources::Compress</term>
 210       <listitem><para>
 211       This is similar to <literal>Packages::Compress</literal>
 212       except that it controls the compression for the Sources files.</para></listitem>
 213       </varlistentry>
 214
 215       <varlistentry><term>Sources::Extensions</term>
 216       <listitem><para>
 217       Sets the default list of file extensions that are source files.
 218       This defaults to '.dsc'.</para></listitem>
 219       </varlistentry>
 220
 221       <varlistentry><term>Contents::Compress</term>
 222       <listitem><para>
 223       This is similar to <literal>Packages::Compress</literal>
 224       except that it controls the compression for the Contents files.</para></listitem>
 225       </varlistentry>
 226
 227       <varlistentry><term>DeLinkLimit</term>
 228       <listitem><para>
 229       Specifies the number of kilobytes to delink (and
 230       replace with hard links) per run. This is used in conjunction with the
 231       per-section <literal>External-Links</literal> setting.</para></listitem>
 232       </varlistentry>
 233
 234       <varlistentry><term>FileMode</term>
 235       <listitem><para>
 236       Specifies the mode of all created index files. It
 237       defaults to 0644. All index files are set to this mode with no regard
 238       to the umask.</para></listitem>
 239       </varlistentry>
 240      </variablelist>
 241    </refsect2>
 242
 243    <refsect2><title>TreeDefault Section</title>
 244      <para>
 245      Sets defaults specific to <literal>Tree</literal> sections. All of these
 246      variables are substitution variables and have the strings $(DIST),
 247      $(SECTION) and $(ARCH) replaced with their respective values.</para>
 248
 249      <variablelist>
 250       <varlistentry><term>MaxContentsChange</term>
 251       <listitem><para>
 252       Sets  the number of kilobytes of contents
 253       files that are generated each day. The contents files are round-robined
 254       so that over several days they will all be rebuilt.</para></listitem>
 255       </varlistentry>
 256
 257       <varlistentry><term>ContentsAge</term>
 258       <listitem><para>
 259       Controls the number of days a contents file is allowed
 260       to be checked without changing. If this limit is passed the mtime of the
 261       contents file is updated. This case can occur if the package file is
 262       changed in such a way that does not result in a new contents file
 263       [override edit for instance]. A hold off is allowed in hopes that new
 264       .debs will be installed, requiring a new file anyhow. The default is 10,
 265       the units are in days.</para></listitem>
 266       </varlistentry>
 267
 268       <varlistentry><term>Directory</term>
 269       <listitem><para>
 270       Sets the top of the .deb directory tree. Defaults to
 271       <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/</filename></para></listitem>
 272       </varlistentry>
 273
 274       <varlistentry><term>SrcDirectory</term>
 275       <listitem><para>
 276       Sets the top of the source package directory tree. Defaults to
 277       <filename>$(DIST)/$(SECTION)/source/</filename></para></listitem>
 278       </varlistentry>
 279
 280       <varlistentry><term>Packages</term>
 281       <listitem><para>
 282       Sets the output Packages file. Defaults to
 283       <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/Packages</filename></para></listitem>
 284       </varlistentry>
 285
 286       <varlistentry><term>Sources</term>
 287       <listitem><para>
 288       Sets the output Sources file. Defaults to
 289       <filename>$(DIST)/$(SECTION)/source/Sources</filename></para></listitem>
 290       </varlistentry>
 291
 292       <varlistentry><term>InternalPrefix</term>
 293       <listitem><para>
 294       Sets the path prefix that causes a symlink to be
 295       considered an internal link instead of an external link. Defaults to
 296       <filename>$(DIST)/$(SECTION)/</filename></para></listitem>
 297       </varlistentry>
 298
 299       <varlistentry><term>Contents</term>
 300       <listitem><para>
 301       Sets the output Contents file. Defaults to
 302       <filename>$(DIST)/Contents-$(ARCH)</filename>. If this setting causes multiple
 303       Packages files to map onto a single Contents file (such as the default)
 304       then <command>apt-ftparchive</command> will integrate those package files
 305       together automatically.</para></listitem>
 306       </varlistentry>
 307
 308       <varlistentry><term>Contents::Header</term>
 309       <listitem><para>
 310       Sets header file to prepend to the contents output.</para></listitem>
 311       </varlistentry>
 312
 313       <varlistentry><term>BinCacheDB</term>
 314       <listitem><para>
 315       Sets the binary cache database to use for this
 316       section. Multiple sections can share the same database.</para></listitem>
 317       </varlistentry>
 318
 319       <varlistentry><term>FileList</term>
 320       <listitem><para>
 321       Specifies that instead of walking the directory tree,
 322       <command>apt-ftparchive</command> should read the list of files from the given
 323       file. Relative files names are prefixed with the archive directory.</para></listitem>
 324       </varlistentry>
 325
 326       <varlistentry><term>SourceFileList</term>
 327       <listitem><para>
 328       Specifies that instead of walking the directory tree,
 329       <command>apt-ftparchive</command> should read the list of files from the given
 330       file. Relative files names are prefixed with the archive directory.
 331       This is used when processing source indexes.</para></listitem>
 332       </varlistentry>
 333      </variablelist>
 334    </refsect2>
 335
 336    <refsect2><title>Tree Section</title>
 337      <para>
 338      The <literal>Tree</literal> section defines a standard Debian file tree which
 339      consists of a base directory, then multiple sections in that base
 340      directory and finally multiple Architectures in each section. The exact
 341      pathing used is defined by the <literal>Directory</literal> substitution variable.</para>
 342      <para>
 343      The <literal>Tree</literal> section takes a scope tag which sets the
 344      <literal>$(DIST)</literal> variable and defines the root of the tree
 345      (the path is prefixed by <literal>ArchiveDir</literal>).
 346      Typically this is a setting such as <filename>dists/woody</filename>.</para>
 347      <para>
 348      All of the settings defined in the <literal>TreeDefault</literal> section can be
 349      use in a <literal>Tree</literal> section as well as three new variables.</para>
 350      <para>
 351      When processing a <literal>Tree</literal> section <command>apt-ftparchive</command>
 352      performs an operation similar to:
 353      <programlisting>
 354 for i in Sections do
 355    for j in Architectures do
 356       Generate for DIST=scope SECTION=i ARCH=j
 357      </programlisting></para>
 358
 359      <variablelist>
 360       <varlistentry><term>Sections</term>
 361       <listitem><para>
 362       This is a space separated list of sections which appear
 363       under the distribution, typically this is something like
 364       <literal>main contrib non-free</literal></para></listitem>
 365       </varlistentry>
 366
 367       <varlistentry><term>Architectures</term>
 368       <listitem><para>
 369       This is a space separated list of all the
 370       architectures that appear under search section. The special architecture
 371       'source' is used to indicate that this tree has a source archive.</para></listitem>
 372       </varlistentry>
 373
 374       <varlistentry><term>BinOverride</term>
 375       <listitem><para>
 376       Sets the binary override file. The override file
 377       contains section, priority and maintainer address information.</para></listitem>
 378       </varlistentry>
 379
 380       <varlistentry><term>SrcOverride</term>
 381       <listitem><para>
 382       Sets the source override file. The override file
 383       contains section information.</para></listitem>
 384       </varlistentry>
 385
 386       <varlistentry><term>ExtraOverride</term>
 387       <listitem><para>
 388       Sets the binary extra override file.</para></listitem>
 389       </varlistentry>
 390
 391       <varlistentry><term>SrcExtraOverride</term>
 392       <listitem><para>
 393       Sets the source extra override file.</para></listitem>
 394       </varlistentry>
 395      </variablelist>
 396    </refsect2>
 397
 398    <refsect2><title>BinDirectory Section</title>
 399      <para>
 400      The <literal>bindirectory</literal> section defines a binary directory tree
 401      with no special structure. The scope tag specifies the location of
 402      the binary directory and the settings are similar to the <literal>Tree</literal>
 403      section with no substitution variables or
 404      <literal>Section</literal><literal>Architecture</literal> settings.</para>
 405      <variablelist>
 406       <varlistentry><term>Packages</term>
 407       <listitem><para>
 408       Sets the Packages file output.</para></listitem>
 409       </varlistentry>
 410
 411       <varlistentry><term>Sources</term>
 412       <listitem><para>
 413       Sets the Sources file output. At least one of
 414       <literal>Packages</literal> or <literal>Sources</literal> is required.</para></listitem>
 415       </varlistentry>
 416
 417       <varlistentry><term>Contents</term>
 418       <listitem><para>
 419       Sets the Contents file output. (optional)</para></listitem>
 420       </varlistentry>
 421
 422       <varlistentry><term>BinOverride</term>
 423       <listitem><para>
 424       Sets the binary override file.</para></listitem>
 425       </varlistentry>
 426
 427       <varlistentry><term>SrcOverride</term>
 428       <listitem><para>
 429       Sets the source override file.</para></listitem>
 430       </varlistentry>
 431
 432       <varlistentry><term>ExtraOverride</term>
 433       <listitem><para>
 434       Sets the binary extra override file.</para></listitem>
 435       </varlistentry>
 436
 437       <varlistentry><term>SrcExtraOverride</term>
 438       <listitem><para>
 439       Sets the source extra override file.</para></listitem>
 440       </varlistentry>
 441
 442       <varlistentry><term>BinCacheDB</term>
 443       <listitem><para>
 444       Sets the cache DB.</para></listitem>
 445       </varlistentry>
 446
 447       <varlistentry><term>PathPrefix</term>
 448       <listitem><para>
 449       Appends a path to all the output paths.</para></listitem>
 450       </varlistentry>
 451
 452       <varlistentry><term>FileList, SourceFileList</term>
 453       <listitem><para>
 454       Specifies the file list file.</para></listitem>
 455       </varlistentry>
 456      </variablelist>
 457    </refsect2>
 458  </refsect1>
 459
 460
 461  <refsect1><title>The Binary Override File</title>
 462    <para>The binary override file is fully compatible with &dpkg-scanpackages;. It
 463    contains 4 fields separated by spaces. The first field is the package name,
 464    the second is the priority to force that package to, the third is the
 465    the section to force that package to and the final field is the maintainer
 466    permutation field.</para>
 467    <para>The general form of the maintainer field is:
 468    <literallayout>old [// oldn]* => new</literallayout>
 469    or simply,
 470    <literallayout>new</literallayout>
 471    The first form allows a double-slash separated list of old email addresses
 472    to be specified. If any of those are found then new is substituted for the
 473    maintainer field. The second form unconditionally substitutes the
 474    maintainer field.</para>
 475  </refsect1>
 476
 477
 478  <refsect1><title>The Source Override File</title>
 479    <para>
 480    The source override file is fully compatible with &dpkg-scansources;. It
 481    contains 2 fields separated by spaces. The first fields is the source
 482    package name, the second is the section to assign it.</para>
 483  </refsect1>
 484
 485  <refsect1><title>The Extra Override File</title>
 486    <para>
 487    The extra override file allows any arbitrary tag to be added or replaced
 488    in the output. It has 3 columns, the first is the package, the second is
 489    the tag and the remainder of the line is the new value.</para>
 490  </refsect1>
 491
 492  <refsect1><title>options</title>
 493    &apt-cmdblurb;
 494
 495    <variablelist>
 496      <varlistentry><term><option>--md5</option></term>
 497      <listitem><para>
 498      Generate MD5 sums. This defaults to on, when turned off the generated
 499      index files will not have MD5Sum fields where possible.
 500      Configuration Item: <literal>APT::FTPArchive::MD5</literal></para></listitem>
 501      </varlistentry>
 502
 503      <varlistentry><term><option>-d</option></term><term><option>--db</option></term>
 504      <listitem><para>
 505      Use a binary caching DB. This has no effect on the generate command.
 506      Configuration Item: <literal>APT::FTPArchive::DB</literal>.</para></listitem>
 507      </varlistentry>
 508
 509      <varlistentry><term><option>-q</option></term><term><option>--quiet</option></term>
 510      <listitem><para>
 511      Quiet; produces output suitable for logging, omitting progress indicators.
 512      More q's will produce more quiet up to a maximum of 2. You can also use
 513      <option>-q=#</option> to set the quiet level, overriding the configuration file.
 514      Configuration Item: <literal>quiet</literal>.</para></listitem>
 515      </varlistentry>
 516
 517      <varlistentry><term><option>--delink</option></term>
 518      <listitem><para>
 519      Perform Delinking. If the <literal>External-Links</literal> setting is used then
 520      this option actually enables delinking of the files. It defaults to on and
 521      can be turned off with <option>--no-delink</option>.
 522      Configuration Item: <literal>APT::FTPArchive::DeLinkAct</literal>.</para></listitem>
 523      </varlistentry>
 524
 525      <varlistentry><term><option>--contents</option></term>
 526      <listitem><para>
 527      Perform contents generation. When this option is set and package indexes
 528      are being generated with a cache DB then the file listing will also be
 529      extracted and stored in the DB for later use. When using the generate
 530      command this option also allows the creation of any Contents files. The
 531      default is on.
 532      Configuration Item: <literal>APT::FTPArchive::Contents</literal>.</para></listitem>
 533      </varlistentry>
 534
 535      <varlistentry><term><option>-s</option></term><term><option>--source-override</option></term>
 536      <listitem><para>
 537      Select the source override file to use with the <literal>sources</literal> command.
 538      Configuration Item: <literal>APT::FTPArchive::SourceOverride</literal>.</para></listitem>
 539      </varlistentry>
 540
 541      <varlistentry><term><option>--readonly</option></term>
 542      <listitem><para>
 543      Make the caching databases read only.
 544      Configuration Item: <literal>APT::FTPArchive::ReadOnlyDB</literal>.</para></listitem>
 545      </varlistentry>
 546
 547      <varlistentry><term><option>APT::FTPArchive::LongDescription</option></term>
 548      <listitem><para>
 549      This configuration option defaults to "<literal>true</literal>" and should only be set to
 550      <literal>"false"</literal> if the Archive generated with &apt-ftparchive; also provides
 551      <filename>Translation</filename> files. Note that it is currently not possible to create these
 552      files with <command>apt-ftparchive</command>.
 553      </para></listitem>
 554      </varlistentry>
 555
 556      &apt-commonoptions;
 557
 558    </variablelist>
 559  </refsect1>
 560
 561 <refsect1><title>Examples</title>
 562
 563 <para>To create a compressed Packages file for a directory containing
 564 binary packages (.deb):
 565
 566 <programlisting>
 567 <command>apt-ftparchive</command> packages <replaceable>directory</replaceable> | <command>gzip</command> > <filename>Packages.gz</filename>
 568 </programlisting></para>
 569
 570 </refsect1>
 571
 572  <refsect1><title>See Also</title>
 573    <para>&apt-conf;</para>
 574  </refsect1>
 575
 576  <refsect1><title>Diagnostics</title>
 577    <para><command>apt-ftparchive</command> returns zero on normal operation, decimal 100 on error.</para>
 578  </refsect1>
 579
 580  &manbugs;
 581
 582 </refentry>