To resolve dependencies like "pkg:arch" we create a package with the
name "pkg:arch" and the architecture "any". We create these packages
only if a dependency needs it as these kind of dependencies aren't that
common. This commit ensured that in the even this architecture specific
dependency is the only relation this package has we still create the
underlying package to have them available in provides resolution.
I only looked at parameters in the previous commit, which was
not enough: One place also generated local string views. In this
case, we only need to make ArchA dynamic, as NameA is not used
after the FindPkg() call.
Remap StringView instances pointing into the cache
It turns out that StringViews might need to be remapped in some
places because they come from the cache. For example, some sites
pass a Ver.VerStr() to NewProvides().
Such a StringView would become invalid during the duration of
the call if the cache is remapped, causing the program to die
with a segmentation fault.
We can take care of those issues by remapping string views in
the same way we remap all the iterators. String views are only
remapped if they point into the cache though, this allows us
to write more generic code on the callee site without having
to check whether the view points into the cache or not.
That's not as efficient as possible, but the overhead does not
appear to be measurable.
string_view: Drop constexpr constructor for standard compatibility
APT::StringView is supposed to be a temporary measure, until support
for the standardized string_view is widely available. Introducing
additional unstandardized features just makes porting to the
standard version harder.
The constexpr constructor also won't have any real effect on most
systems, as the compiler will happily optimise the strlen() call
away for constant strings.
return correct position in APT::StringView::(r)find
The position returned is supposed to be the position of the character
counted from the start of the string, but if we used the substr calling
overloads the skipped over prefix wasn't considered. The pos parameter
of rfind had also the wrong semantic.
Introduced in 9d2a8a7388cf3b0bbbe92f6b0b30a533e1167f40 apt tries to
merge actions like downloading the same (as judged by hashes) file
into doing it once. The implementation was very simple in that it isn't
planing at all. Turns out that it works 90% of the time just fine, but
has issues in more complicated situations in which items can be in
different stages downloading different files emitting potentially the
"wrong" hash – like while pdiffs are worked on we might end up copying
the patch instead of the result file giving us very strange errors in
return. Reverting the change until we can implement a better planing
solution seems to be the best course of action even if its sad.
fix M-A:foreign provides creation for unknown archs
Architectures for packages which do not belong to the native nor a
foreign architecture (dubbed barbarian for now) which are marked
M-A:foreign still provide in their own architecture even if not for
others. Also, other M-A:foreign (and allowed) packages provide in these
barbarian architectures.
evaluate sourceslist-list-format entity in vendors sources.list
Parsing XML entity files in shell isn't exactly nice and causing the
substitution with a while-read loop isn't either, but it seems to be
good enough for the moment without changing too much.
If a package has no description, we would crash in search. While
this should not happen, there seem to be some weird cases where
it does.
A safer way might be to make the whole parser thing safe
against this, so pkgRecords::Lookup(Desc.FileList()) works
and returns a parser where all values are empty. This would
also fix all other instances of this bug, if there are any.
remove uncompressed leftover partial file before pdiff bootstrap
The code already deals with compressed leftovers, but forgot the
uncompressed files. The opertunity is picked to reorder this code and
add debug messages about the actions taken as well as produce such a
leftover file in the associated testcase.
use filesize of compressed pdiffs for the limit if possible
With the addition of the $HASH-Download field in the .diff/Index we got
the size of the compressed patches for 'free', so if that information is
available we can use it for a more fitting calculation of the size
requirements of the patches vs. the complete file.
Note that this predicts a too small size in the transition case in which
the information isn't available for all patches, but figuring this out
would be a lot of code for practically nothing as only one update can
ever be in such a transition phase.
tests: limit autotest-functionname generation to sane characters
Some (older) versions of bash seem to be allergic to a method named
"aptautotest_grep_^apt" (note the caret). Unlikely that we are going to
write autotests for such commands so we could just skip those, but lets
instead just use "normal" characters in the names and strip the rest as
we already did with the (arguable more common) '-'.
support '-' and no parameter for stdin in apt-helper cat-file
This way it works more similar to the compressor binaries, which we
can relief in this way from their job in the test framework avoiding the
need of adding e.g. liblz4-tool to the test dependencies.
Downloading and storing are two different operations were different
compression types can be preferred. For downloading we provide the
choice via Acquire::CompressionTypes::Order as there is a choice to
be made between download size and speed – and limited by whats available
in the repository.
Storage on the other hand has all compressions currently supported by
apt available and to reduce runtime of tools accessing these files the
compression type should be a low-cost format in terms of decompression.
apt traditionally stores its indexes uncompressed on disk, but has
options to keep them compressed. Now that apt downloads additional files
we also deal with files which simply can't be stored uncompressed as
they are just too big (like Contents for apt-file). Traditionally they
are downloaded in a low-cost format (gz) as repositories do not provide
other formats, but there might be even lower-cost formats and for
download we could introduce higher-cost in the repositories.
Downloading an entire index potentially requires recompression to
another format, so an update takes potentially longer – but big files
are usually updated via pdiffs which has to de- and re-compress anyhow
and does it on the fly anyhow, so there is no extra time needed and in
general it seems to be benefitial to invest the time in update to save
time later on file access.
allow pdiff bootstrap from all supported compressors
There is no reason to enforce that the file we start the bootstrap with
is compressed with a compressor which is available online. This allows
us to change the on-disk format as well as deals with repositories
adding/removing support for a specific compressor.
ensure compression cleanup even without lists-cleanup
If we store files compressed in lists/ and the file switched compression
formats we happened to retain the "old" format, but by default the
cleanup process catched this oversight and removed the file.
[The initial situation described doesn't arise as we store no files by
default compressed and even with apt-file configuring Contents files, we
don't really have that problem as there is just .gz files for those.]
We solve this by just removing any uncompressed as well as compressed
(we support) file just before we move the 'new' version of the file in.
use one 'store' method to rule all (de)compressors
Adding a new compressor method meant adding a new method as well – even
if that boilt down to just linking to our generalized decompressor with
a new name. That is unneeded busywork if we can instead just call the
generalized decompressor and let it figure out which compressor to use
based on the filenames rather than by program name.
For compatibility we ship still 'gzip', 'bzip2' and co, but they are
just links to our "new" 'store' method.
include all compressed Packages/Sources files in Release file
Having a hardcoded list of compression types here doesn't really provide
us with anything beside added complexity each time someone adds a new
compression type. That we don't need to be that specific is evident by
Contents and Translation-* matchers which are a lot more generic and
didn't generate problems anyhow.
AvailableDescriptionLanguages: Use one string for all iterations
Do not create strings within the loop, that creates one string
per language and does more work than needed. Instead, reserve
enough space at the beginning and assign the prefix, and then
resize and append inside the loop.
Also call exists with the string itself instead of the c_str(),
this means that the lookup uses the size information in the
string now and does not have to call strlen() on it.
pkgCacheGenerator::StoreString: Get rid of std::string
Instead of storing a string -> map_stringitem_t mapping, create
our own data type that can point to either a normal string or
a string inside the cache.
This avoids the creation of any string and improves performance
slightly (about 4%).
pkgCacheGenerator::StoreString: Move the string into the map
Moving the string is likely faster than copying it. We could probably
avoid strings alltogether in the future using some more crazy code,
but I have not looked at that yet.
The class APT::StringView implements a drop-in replacement
for a subset of C++17 std::string_view() features. It will
be dropped at a later point and may not be used in public
interfaces.
acquire: Allow parallelizing methods without hosts
The maximum parallelization soft limit is the number of CPU
cores * 2 on systems defining _SC_NPROCESSORS_ONLN. The hard
limit in all cases is Acquire::QueueHost::Limit.
Helmut Grohne [Sun, 3 Jan 2016 14:10:05 +0000 (15:10 +0100)]
Allow building without libgtest-dev under <nocheck> build profile
I'd like to avoid pulling libgtest-dev into the bootstrap set.
Fortunately, libgtest-dev is only used for testing apt and apt
correctly implements DEB_BUILD_OPTIONS=nocheck now. So this
bug is about getting rid of the Build-Depends.
Simply removing it (by adding a build profile) is not sufficient
however as configure fails hard, so an additional bit is necessary
to cover for that.
Michael Vogt [Sat, 2 Jan 2016 21:08:30 +0000 (22:08 +0100)]
Add new APT::Keep-Downloaded-Packages option
This option controls if downloaded packages should be kept after
a successful install or if they should be deleted. The default
for "apt-get" is that they are kept (just like before).
However the default for "apt" is that they get deleted.
Michael Vogt [Sat, 2 Jan 2016 20:18:47 +0000 (21:18 +0100)]
Add bash-completion support for the "apt" command
The apt bash-completion support was submited to the bash-completion
package as a patch in May 2014. It is still not included to this
date and because it is an important feature for many users it is
now part of apt until the bash-completion package is mantained
more actively again.
Note that the "Relaces" line is only required for Ubuntu it will
have no effect on Debian.
Michael Vogt [Sat, 2 Jan 2016 20:24:47 +0000 (21:24 +0100)]
Add support for APT::Periodic::CleanInterval
In order to allow regular cleaning of the downloaded deb archives
the new option APT::Periodic::CleanInterval is added. It will run
"apt-get clean" in the given time interval.
properly parse comments in apt_preferences and deb822-style sources
apt_preferences and deb822-style sources used the specialized class
pkgUserTagSection to deal with comments before/after a given stanza, but
it couldn't deal with comments in the stanza at all.
codesearch suggests that nobody else does and a vastely superior way of
working with potentially commented files is implemented now, so we can
officially discourage the use of the old incomplete hack class.
Now (55153bf94ff28a23318e79aa48242244c4d82b3c) that pkgTagFile can be
told to deal with all sorts of comments we can use this mode to parse
dsc (as by catch) and debian/control files properly even in the wake of
multiline fields spliced with comments like Build-Depends.
APT usually deals with perfectly formatted files generated automatically
be other programs – and as it has to parse multiple MBs of such files it
tries to be fast rather than forgiving.
This was always a problem if we reused this parser for files with a
deb822 syntax which are mostly written by hand however, like
apt_preferences or the deb822-style sources as these can include stray
newlines and more importantly comments all over the place.
As a stopgap we had pkgUserTagSection which deals at least with comments
before and after a given stanza, but comments in between weren't really
supported and now that we support parsing debian/control for e.g.
build-dep we face the full comment problem e.g. with comments inbetween
multi-line fields (like Build-Depends).
We can't easily deal with this on the pkgTagSection level as the interface
gives access to 'raw' char-pointers for performance reasons so we would
need to optionally add a buffer here on which we could remove comments
to hand out pointers into this buffer instead. The interface is quite
large already and supports writing stanzas as well, which does not
support comments at all either. So while in future it might make sense
to have a parser setup which deals with and keeps comments in this
commit we opt for the simpler solution for now: We officially declare
that pkgTagSection does not support comments and instead expect the
caller to deal with them, which in our case is pkgTagFile:
pkgTagFile is extended with an additional mode which can deal with
comments by dropping them from the buffer which will later form the
input of pkgTagSection. The actual implementation is slightly more
complex than this sentence suggests at first on one hand to have good
performance and on the other to allow jumping directly to stanzas with
offsets collected in a previous run (like our cache generation does it
for example).
projects / apt.git / log
