doc/apt-ftparchive.1.xml

   1 <?xml version="1.0" encoding="utf-8" standalone="no"?>
   2 <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
   3   "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [
   4
   5 <!ENTITY % aptent SYSTEM "apt.ent">
   6 %aptent;
   7
   8 ]>
   9
  10 <refentry>
  11
  12  <refentryinfo>
  13    &apt-author.jgunthorpe;
  14    &apt-author.team;
  15    &apt-email;
  16    &apt-product;
  17    <!-- The last update date -->
  18    <date>29 February 2004</date>
  19  </refentryinfo>
  20
  21  <refmeta>
  22    <refentrytitle>apt-ftparchive</refentrytitle>
  23    <manvolnum>1</manvolnum>
  24  </refmeta>
  25
  26  <!-- Man page title -->
  27  <refnamediv>
  28     <refname>apt-ftparchive</refname>
  29     <refpurpose>Utility to generate index files</refpurpose>
  30  </refnamediv>
  31
  32  <!-- Arguments -->
  33  <refsynopsisdiv>
  34    <cmdsynopsis>
  35       <command>apt-ftparchive</command>
  36       <arg><option>-hvdsq</option></arg>
  37       <arg><option>--md5</option></arg>
  38       <arg><option>--delink</option></arg>
  39       <arg><option>--readonly</option></arg>
  40       <arg><option>--contents</option></arg>
  41       <arg><option>-o=<replaceable>config string</replaceable></option></arg>
  42       <arg><option>-c=<replaceable>file</replaceable></option></arg>
  43       <group choice="req">
  44          <arg>packages<arg choice="plain" rep="repeat"><replaceable>path</replaceable></arg><arg><replaceable>override</replaceable><arg><replaceable>pathprefix</replaceable></arg></arg></arg>
  45          <arg>sources<arg choice="plain" rep="repeat"><replaceable>path</replaceable></arg><arg><replaceable>override</replaceable><arg><replaceable>pathprefix</replaceable></arg></arg></arg>
  46          <arg>contents <arg choice="plain"><replaceable>path</replaceable></arg></arg>
  47          <arg>release <arg choice="plain"><replaceable>path</replaceable></arg></arg>
  48          <arg>generate <arg choice="plain"><replaceable>config-file</replaceable></arg> <arg choice="plain" rep="repeat"><replaceable>section</replaceable></arg></arg>
  49          <arg>clean <arg choice="plain"><replaceable>config-file</replaceable></arg></arg>
  50       </group>
  51    </cmdsynopsis>
  52  </refsynopsisdiv>
  53
  54  <refsect1><title>Description</title>
  55    <para><command>apt-ftparchive</command> is the command line tool that generates the index
  56    files that APT uses to access a distribution source. The index files should
  57    be generated on the origin site based on the content of that site.</para>
  58
  59    <para><command>apt-ftparchive</command> is a superset of the &dpkg-scanpackages; program,
  60    incorporating its entire functionality via the <literal>packages</literal> command.
  61    It also contains a contents file generator, <literal>contents</literal>, and an
  62    elaborate means to 'script' the generation process for a complete
  63    archive.</para>
  64
  65    <para>Internally <command>apt-ftparchive</command> can make use of binary databases to
  66    cache the contents of a .deb file and it does not rely on any external
  67    programs aside from &gzip;. When doing a full generate it automatically
  68    performs file-change checks and builds the desired compressed output files.</para>
  69
  70    <para>Unless the <option>-h</option>, or <option>--help</option> option is given one of the
  71    commands below must be present.</para>
  72
  73    <variablelist>
  74      <varlistentry><term>packages</term>
  75      <listitem><para>
  76      The packages command generates a package file from a directory tree. It
  77      takes the given directory and recursively searches it for .deb files,
  78      emitting a package record to stdout for each. This command is
  79      approximately equivalent to &dpkg-scanpackages;.</para>
  80
  81      <para>The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
  82      </varlistentry>
  83
  84      <varlistentry><term>sources</term>
  85      <listitem><para>
  86      The <literal>sources</literal> command generates a source index file from a directory tree.
  87      It takes the given directory and recursively searches it for .dsc files,
  88      emitting a source record to stdout for each. This command is approximately
  89      equivalent to &dpkg-scansources;.</para>
  90      <para>
  91      If an override file is specified then a source override file will be
  92      looked for with an extension of .src. The --source-override option can be
  93      used to change the source override file that will be used.</para></listitem>
  94      </varlistentry>
  95
  96      <varlistentry><term>contents</term>
  97      <listitem><para>
  98      The <literal>contents</literal> command generates a contents file from a directory tree. It
  99      takes the given directory and recursively searches it for .deb files,
 100      and reads the file list from each file. It then sorts and writes to stdout
 101      the list of files matched to packages. Directories are not written to
 102      the output. If multiple packages own the same file then each package is
 103      separated by a comma in the output.</para>
 104      <para>
 105      The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
 106      </varlistentry>
 107
 108      <varlistentry><term>release</term>
 109      <listitem><para>
 110      The <literal>release</literal> command generates a Release file from a
 111      directory tree. It recursively searches the given directory for
 112      Packages, Packages.gz, Packages.bz2, Sources, Sources.gz,
 113      Sources.bz2, Release and md5sum.txt files.  It then writes to
 114      stdout a Release file containing an MD5 digest and SHA1 digest
 115      for each file.</para>
 116      <para>
 117      Values for the additional metadata fields in the Release file are
 118      taken from the corresponding variables under
 119      <literal>APT::FTPArchive::Release</literal>,
 120      e.g. <literal>APT::FTPArchive::Release::Origin</literal>.  The supported fields
 121      are: <literal>Origin</literal>, <literal>Label</literal>, <literal>Suite</literal>,
 122      <literal>Version</literal>, <literal>Codename</literal>, <literal>Date</literal>,
 123      <literal>Architectures</literal>, <literal>Components</literal>, <literal>Description</literal>.</para></listitem>
 124
 125      </varlistentry>
 126
 127      <varlistentry><term>generate</term>
 128      <listitem><para>
 129      The <literal>generate</literal> command is designed to be runnable from a cron script and
 130      builds indexes according to the given config file. The config language
 131      provides a flexible means of specifying which index files are built from
 132      which directories, as well as providing a simple means of maintaining the
 133      required settings.</para></listitem>
 134      </varlistentry>
 135
 136      <varlistentry><term>clean</term>
 137      <listitem><para>
 138      The <literal>clean</literal> command tidies the databases used by the given
 139      configuration file by removing any records that are no longer necessary.</para></listitem>
 140      </varlistentry>
 141    </variablelist>
 142  </refsect1>
 143
 144  <refsect1><title>The Generate Configuration</title>
 145    <para>
 146    The <literal>generate</literal> command uses a configuration file to describe the
 147    archives that are going to be generated. It follows the typical ISC
 148    configuration format as seen in ISC tools like bind 8 and dhcpd.
 149    &apt-conf; contains a description of the syntax. Note that the generate
 150    configuration is parsed in sectional manner, but &apt-conf; is parsed in a
 151    tree manner. This only effects how the scope tag is handled.</para>
 152
 153    <para>
 154    The generate configuration has 4 separate sections, each described below.</para>
 155
 156    <refsect2><title>Dir Section</title>
 157      <para>
 158      The <literal>Dir</literal> section defines the standard directories needed to
 159      locate the files required during the generation process. These
 160      directories are prepended to certain relative paths defined in later
 161      sections to produce a complete an absolute path.</para>
 162      <variablelist>
 163       <varlistentry><term>ArchiveDir</term>
 164       <listitem><para>
 165       Specifies the root of the FTP archive, in a standard
 166       Debian configuration this is the directory that contains the
 167       <filename>ls-LR</filename> and dist nodes.</para></listitem>
 168       </varlistentry>
 169
 170       <varlistentry><term>OverrideDir</term>
 171       <listitem><para>
 172       Specifies the location of the override files.</para></listitem>
 173       </varlistentry>
 174
 175       <varlistentry><term>CacheDir</term>
 176       <listitem><para>
 177       Specifies the location of the cache files</para></listitem>
 178       </varlistentry>
 179
 180       <varlistentry><term>FileListDir</term>
 181       <listitem><para>
 182       Specifies the location of the file list files,
 183       if the <literal>FileList</literal> setting is used below.</para></listitem>
 184       </varlistentry>
 185      </variablelist>
 186    </refsect2>
 187
 188    <refsect2><title>Default Section</title>
 189      <para>
 190      The <literal>Default</literal> section specifies default values, and settings
 191      that control the operation of the generator. Other sections may override
 192      these defaults with a per-section setting.</para>
 193      <variablelist>
 194       <varlistentry><term>Packages::Compress</term>
 195       <listitem><para>
 196       Sets the default compression schemes to use
 197       for the Package index files. It is a string that contains a space
 198       separated list of at least one of: '.' (no compression), 'gzip' and
 199       'bzip2'. The default for all compression schemes is '. gzip'.</para></listitem>
 200       </varlistentry>
 201
 202       <varlistentry><term>Packages::Extensions</term>
 203       <listitem><para>
 204       Sets the default list of file extensions that are package files.
 205       This defaults to '.deb'.</para></listitem>
 206       </varlistentry>
 207
 208       <varlistentry><term>Sources::Compress</term>
 209       <listitem><para>
 210       This is similar to <literal>Packages::Compress</literal>
 211       except that it controls the compression for the Sources files.</para></listitem>
 212       </varlistentry>
 213
 214       <varlistentry><term>Sources::Extensions</term>
 215       <listitem><para>
 216       Sets the default list of file extensions that are source files.
 217       This defaults to '.dsc'.</para></listitem>
 218       </varlistentry>
 219
 220       <varlistentry><term>Contents::Compress</term>
 221       <listitem><para>
 222       This is similar to <literal>Packages::Compress</literal>
 223       except that it controls the compression for the Contents files.</para></listitem>
 224       </varlistentry>
 225
 226       <varlistentry><term>DeLinkLimit</term>
 227       <listitem><para>
 228       Specifies the number of kilobytes to delink (and
 229       replace with hard links) per run. This is used in conjunction with the
 230       per-section <literal>External-Links</literal> setting.</para></listitem>
 231       </varlistentry>
 232
 233       <varlistentry><term>FileMode</term>
 234       <listitem><para>
 235       Specifies the mode of all created index files. It
 236       defaults to 0644. All index files are set to this mode with no regard
 237       to the umask.</para></listitem>
 238       </varlistentry>
 239      </variablelist>
 240    </refsect2>
 241
 242    <refsect2><title>TreeDefault Section</title>
 243      <para>
 244      Sets defaults specific to <literal>Tree</literal> sections. All of these
 245      variables are substitution variables and have the strings $(DIST),
 246      $(SECTION) and $(ARCH) replaced with their respective values.</para>
 247
 248      <variablelist>
 249       <varlistentry><term>MaxContentsChange</term>
 250       <listitem><para>
 251       Sets  the number of kilobytes of contents
 252       files that are generated each day. The contents files are round-robined
 253       so that over several days they will all be rebuilt.</para></listitem>
 254       </varlistentry>
 255
 256       <varlistentry><term>ContentsAge</term>
 257       <listitem><para>
 258       Controls the number of days a contents file is allowed
 259       to be checked without changing. If this limit is passed the mtime of the
 260       contents file is updated. This case can occur if the package file is
 261       changed in such a way that does not result in a new contents file
 262       [override edit for instance]. A hold off is allowed in hopes that new
 263       .debs will be installed, requiring a new file anyhow. The default is 10,
 264       the units are in days.</para></listitem>
 265       </varlistentry>
 266
 267       <varlistentry><term>Directory</term>
 268       <listitem><para>
 269       Sets the top of the .deb directory tree. Defaults to
 270       <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/</filename></para></listitem>
 271       </varlistentry>
 272
 273       <varlistentry><term>SrcDirectory</term>
 274       <listitem><para>
 275       Sets the top of the source package directory tree. Defaults to
 276       <filename>$(DIST)/$(SECTION)/source/</filename></para></listitem>
 277       </varlistentry>
 278
 279       <varlistentry><term>Packages</term>
 280       <listitem><para>
 281       Sets the output Packages file. Defaults to
 282       <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/Packages</filename></para></listitem>
 283       </varlistentry>
 284
 285       <varlistentry><term>Sources</term>
 286       <listitem><para>
 287       Sets the output Packages file. Defaults to
 288       <filename>$(DIST)/$(SECTION)/source/Sources</filename></para></listitem>
 289       </varlistentry>
 290
 291       <varlistentry><term>InternalPrefix</term>
 292       <listitem><para>
 293       Sets the path prefix that causes a symlink to be
 294       considered an internal link instead of an external link. Defaults to
 295       <filename>$(DIST)/$(SECTION)/</filename></para></listitem>
 296       </varlistentry>
 297
 298       <varlistentry><term>Contents</term>
 299       <listitem><para>
 300       Sets the output Contents file. Defaults to
 301       <filename>$(DIST)/Contents-$(ARCH)</filename>. If this setting causes multiple
 302       Packages files to map onto a single Contents file (such as the default)
 303       then <command>apt-ftparchive</command> will integrate those package files
 304       together automatically.</para></listitem>
 305       </varlistentry>
 306
 307       <varlistentry><term>Contents::Header</term>
 308       <listitem><para>
 309       Sets header file to prepend to the contents output.</para></listitem>
 310       </varlistentry>
 311
 312       <varlistentry><term>BinCacheDB</term>
 313       <listitem><para>
 314       Sets the binary cache database to use for this
 315       section. Multiple sections can share the same database.</para></listitem>
 316       </varlistentry>
 317
 318       <varlistentry><term>FileList</term>
 319       <listitem><para>
 320       Specifies that instead of walking the directory tree,
 321       <command>apt-ftparchive</command> should read the list of files from the given
 322       file. Relative files names are prefixed with the archive directory.</para></listitem>
 323       </varlistentry>
 324
 325       <varlistentry><term>SourceFileList</term>
 326       <listitem><para>
 327       Specifies that instead of walking the directory tree,
 328       <command>apt-ftparchive</command> should read the list of files from the given
 329       file. Relative files names are prefixed with the archive directory.
 330       This is used when processing source indexs.</para></listitem>
 331       </varlistentry>
 332      </variablelist>
 333    </refsect2>
 334
 335    <refsect2><title>Tree Section</title>
 336      <para>
 337      The <literal>Tree</literal> section defines a standard Debian file tree which
 338      consists of a base directory, then multiple sections in that base
 339      directory and finally multiple Architectures in each section. The exact
 340      pathing used is defined by the <literal>Directory</literal> substitution variable.</para>
 341      <para>
 342      The <literal>Tree</literal> section takes a scope tag which sets the
 343      <literal>$(DIST)</literal> variable and defines the root of the tree
 344      (the path is prefixed by <literal>ArchiveDir</literal>).
 345      Typically this is a setting such as <filename>dists/woody</filename>.</para>
 346      <para>
 347      All of the settings defined in the <literal>TreeDefault</literal> section can be
 348      use in a <literal>Tree</literal> section as well as three new variables.</para>
 349      <para>
 350      When processing a <literal>Tree</literal> section <command>apt-ftparchive</command>
 351      performs an operation similar to:
 352 <informalexample><programlisting>
 353 for i in Sections do
 354    for j in Architectures do
 355       Generate for DIST=scope SECTION=i ARCH=j
 356 </programlisting></informalexample></para>
 357
 358      <variablelist>
 359       <varlistentry><term>Sections</term>
 360       <listitem><para>
 361       This is a space separated list of sections which appear
 362       under the distribution, typically this is something like
 363       <literal>main contrib non-free</literal></para></listitem>
 364       </varlistentry>
 365
 366       <varlistentry><term>Architectures</term>
 367       <listitem><para>
 368       This is a space separated list of all the
 369       architectures that appear under search section. The special architecture
 370       'source' is used to indicate that this tree has a source archive.</para></listitem>
 371       </varlistentry>
 372
 373       <varlistentry><term>BinOverride</term>
 374       <listitem><para>
 375       Sets the binary override file. The override file
 376       contains section, priority and maintainer address information.</para></listitem>
 377       </varlistentry>
 378
 379       <varlistentry><term>SrcOverride</term>
 380       <listitem><para>
 381       Sets the source override file. The override file
 382       contains section information.</para></listitem>
 383       </varlistentry>
 384
 385       <varlistentry><term>ExtraOverride</term>
 386       <listitem><para>
 387       Sets the binary extra override file.</para></listitem>
 388       </varlistentry>
 389
 390       <varlistentry><term>SrcExtraOverride</term>
 391       <listitem><para>
 392       Sets the source extra override file.</para></listitem>
 393       </varlistentry>
 394      </variablelist>
 395    </refsect2>
 396
 397    <refsect2><title>BinDirectory Section</title>
 398      <para>
 399      The <literal>bindirectory</literal> section defines a binary directory tree
 400      with no special structure. The scope tag specifies the location of
 401      the binary directory and the settings are similar to the <literal>Tree</literal>
 402      section with no substitution variables or
 403      <literal>Section</literal><literal>Architecture</literal> settings.</para>
 404      <variablelist>
 405       <varlistentry><term>Packages</term>
 406       <listitem><para>
 407       Sets the Packages file output.</para></listitem>
 408       </varlistentry>
 409
 410       <varlistentry><term>Sources</term>
 411       <listitem><para>
 412       Sets the Sources file output. At least one of
 413       <literal>Packages</literal> or <literal>Sources</literal> is required.</para></listitem>
 414       </varlistentry>
 415
 416       <varlistentry><term>Contents</term>
 417       <listitem><para>
 418       Sets the Contents file output. (optional)</para></listitem>
 419       </varlistentry>
 420
 421       <varlistentry><term>BinOverride</term>
 422       <listitem><para>
 423       Sets the binary override file.</para></listitem>
 424       </varlistentry>
 425
 426       <varlistentry><term>SrcOverride</term>
 427       <listitem><para>
 428       Sets the source override file.</para></listitem>
 429       </varlistentry>
 430
 431       <varlistentry><term>ExtraOverride</term>
 432       <listitem><para>
 433       Sets the binary extra override file.</para></listitem>
 434       </varlistentry>
 435
 436       <varlistentry><term>SrcExtraOverride</term>
 437       <listitem><para>
 438       Sets the source extra override file.</para></listitem>
 439       </varlistentry>
 440
 441       <varlistentry><term>BinCacheDB</term>
 442       <listitem><para>
 443       Sets the cache DB.</para></listitem>
 444       </varlistentry>
 445
 446       <varlistentry><term>PathPrefix</term>
 447       <listitem><para>
 448       Appends a path to all the output paths.</para></listitem>
 449       </varlistentry>
 450
 451       <varlistentry><term>FileList, SourceFileList</term>
 452       <listitem><para>
 453       Specifies the file list file.</para></listitem>
 454       </varlistentry>
 455      </variablelist>
 456    </refsect2>
 457  </refsect1>
 458
 459
 460  <refsect1><title>The Binary Override File</title>
 461    <para>The binary override file is fully compatible with &dpkg-scanpackages;. It
 462    contains 4 fields separated by spaces. The first field is the package name,
 463    the second is the priority to force that package to, the third is the
 464    the section to force that package to and the final field is the maintainer
 465    permutation field.</para>
 466    <para>The general form of the maintainer field is:
 467    <literallayout>old [// oldn]* => new</literallayout>
 468    or simply,
 469    <literallayout>new</literallayout>
 470    The first form allows a double-slash separated list of old email addresses
 471    to be specified. If any of those are found then new is substituted for the
 472    maintainer field. The second form unconditionally substitutes the
 473    maintainer field.</para>
 474  </refsect1>
 475
 476
 477  <refsect1><title>The Source Override File</title>
 478    <para>
 479    The source override file is fully compatible with &dpkg-scansources;. It
 480    contains 2 fields separated by spaces. The first fields is the source
 481    package name, the second is the section to assign it.</para>
 482  </refsect1>
 483
 484  <refsect1><title>The Extra Override File</title>
 485    <para>
 486    The extra override file allows any arbitrary tag to be added or replaced
 487    in the output. It has 3 columns, the first is the package, the second is
 488    the tag and the remainder of the line is the new value.</para>
 489  </refsect1>
 490
 491  <refsect1><title>options</title>
 492    &apt-cmdblurb;
 493
 494    <variablelist>
 495      <varlistentry><term><option>--md5</option></term>
 496      <listitem><para>
 497      Generate MD5 sums. This defaults to on, when turned off the generated
 498      index files will not have MD5Sum fields where possible.
 499      Configuration Item: <literal>APT::FTPArchive::MD5</literal></para></listitem>
 500      </varlistentry>
 501
 502      <varlistentry><term><option>-d</option></term><term><option>--db</option></term>
 503      <listitem><para>
 504      Use a binary caching DB. This has no effect on the generate command.
 505      Configuration Item: <literal>APT::FTPArchive::DB</literal>.</para></listitem>
 506      </varlistentry>
 507
 508      <varlistentry><term><option>-q</option></term><term><option>--quiet</option></term>
 509      <listitem><para>
 510      Quiet; produces output suitable for logging, omitting progress indicators.
 511      More q's will produce more quiet up to a maximum of 2. You can also use
 512      <option>-q=#</option> to set the quiet level, overriding the configuration file.
 513      Configuration Item: <literal>quiet</literal>.</para></listitem>
 514      </varlistentry>
 515
 516      <varlistentry><term><option>--delink</option></term>
 517      <listitem><para>
 518      Perform Delinking. If the <literal>External-Links</literal> setting is used then
 519      this option actually enables delinking of the files. It defaults to on and
 520      can be turned off with <option>--no-delink</option>.
 521      Configuration Item: <literal>APT::FTPArchive::DeLinkAct</literal>.</para></listitem>
 522      </varlistentry>
 523
 524      <varlistentry><term><option>--contents</option></term>
 525      <listitem><para>
 526      Perform contents generation. When this option is set and package indexes
 527      are being generated with a cache DB then the file listing will also be
 528      extracted and stored in the DB for later use. When using the generate
 529      command this option also allows the creation of any Contents files. The
 530      default is on.
 531      Configuration Item: <literal>APT::FTPArchive::Contents</literal>.</para></listitem>
 532      </varlistentry>
 533
 534      <varlistentry><term><option>-s</option></term><term><option>--source-override</option></term>
 535      <listitem><para>
 536      Select the source override file to use with the <literal>sources</literal> command.
 537      Configuration Item: <literal>APT::FTPArchive::SourceOverride</literal>.</para></listitem>
 538      </varlistentry>
 539
 540      <varlistentry><term><option>--readonly</option></term>
 541      <listitem><para>
 542      Make the caching databases read only.
 543      Configuration Item: <literal>APT::FTPArchive::ReadOnlyDB</literal>.</para></listitem>
 544      </varlistentry>
 545
 546      &apt-commonoptions;
 547
 548    </variablelist>
 549  </refsect1>
 550
 551 <refsect1><title>Examples</title>
 552
 553 <para>To create a compressed Packages file for a directory containing
 554 binary packages (.deb):
 555
 556 <programlisting>
 557 <command>apt-ftparchive</command> packages <replaceable>directory</replaceable> | <command>gzip</command> > <filename>Packages.gz</filename>
 558 </programlisting></para>
 559
 560 </refsect1>
 561
 562  <refsect1><title>See Also</title>
 563    <para>&apt-conf;</para>
 564  </refsect1>
 565
 566  <refsect1><title>Diagnostics</title>
 567    <para><command>apt-ftparchive</command> returns zero on normal operation, decimal 100 on error.</para>
 568  </refsect1>
 569
 570  &manbugs;
 571
 572 </refentry>