Reversing the parsing order ensures that we parse weaker hashes (like
SHA1) before we touch newer/stronger hashes (like SHA256) as the weaker
ones will usually be there for a longer time already with data already
present, which we would discard if we start with the strong one first.
The discarding is visible in the debug logs:
File X wasn't in the list for the first parsed hash! (history)
File X wasn't in the list for the first parsed hash! (patches)
which if file X is part of the patch-path means apt will not find a path and
fallback to acquire the whole file instead needlessly.
If file X isn't part of the patch-path that is no problem, so that
effects only the update-call which updates with patches coming from
before and after the addition of a new hash.
With the package names now normalized to lower case, the caches
of affected systems need to be rebuild. Adjust the minor version
to trigger such a rebuild.
Convert package names from Packages files to lower case
dpkg does that when reading package files, so we should do
the same. This only deals with parsing names from binary
package paragraphs, it does not look at source package names
and/or the list of binaries in a dsc file.
In e75e5879 'replace "which" with "command -v" for portability' I missed
that command -v isn't actually required to be available in debian, so
for the 5 files we are using it:
Two (abicheck/run_abi_test & test/integration/framework) are called in
environments were I believe sh is at least dash or 'better' as the first
one is "interactive" for apt developers and the later is sourced by ~200
tests in the same directory run by hand and ci-services – for the later
we have pulled some uglier hacks for worser things already, so if there
should actually end up needing something more compatible we will notice
eventually (and the later actually had a command -v call for some time
already and nobody came running).
debian/rules and debian/apt.cron.daily I switched back to which as that
is more or less debian-specific or at least highly non-critical.
That leaves cmdline/apt-key.in with a bunch of calls where I will
implement that functionality in shell as this is relatively short-lived
as it is used to detect wget (for net-update, which Michael wants to
revive and in that process will properly use apt-helper instead of wget)
and to detect gpg vs. gpg2 systems, where the earlier is supposed to go
away in the longrun (or the later, but by replacing the earlier…).
[and this gpg/gpg2 detection is new in sid, so I have some sympathy for
that being a problem now.]
replace run-parts with find|sort to avoid debianutils usage
After e75e5879 the reason for an implicit dependency on debianutils
(which is essential for debian, but likely not on other systems) was
just two uses of run-parts, which can be replaced with the a lot more
portable find-piped-into-sort duo.
tests: apt build-dep actually reports no depends correctly
'Regression' of 7d19ee92f2368a40e739cb27d22d6d28f37ebf45, just that it
now works more as expected than previously. Of course, build-essentials
are implicitly also build dependencies, so by definition all packages
have build dependencies, but that isn't what this message wants to say
and it isn't what the user expects.
require explicit paths to dsc/control as we do for deb files
Otherwise a user is subject to unexpected content-injection depending on
which directory she happens to start apt in. This also cleans up the code
requiring less implementation details in build-dep which is always good.
Technically, this is an ABI break as we override virtual methods, but
that they weren't overridden was a mistake resulting in pure classes,
which shouldn't be pure, so they were unusable – and as they are new in
1.1 nobody is using them yet (and hopefully ever as they are borderline
implementation details).
defaults.mak: Set LC_COLLATE=C.UTF-8 for reproducible sort order
With the 1.1.3 release we have seen some re-ordering of the
translation template and the translations. It turns out that
this is because sort sorts differently depending on the locale,
so let's force it to always sort in the C locale.
As we ship some tools in apt-utils which depend on our private library
we have to ensure that apt-utils depends on a proper apt version.
An exact version is probably a bit much, but the simplest way out.
do not override exact targetrelease matches with lesser matches
The relevant testcases are in test/integration/test-apt-get-source.
There is a test for #731853 that is supposed to "ensure that apt will
pick the higher version number" of 0.0.1 (stable) and 0.1 (stable).
However, this works by pure chance, as simply reversing the order
of the two insertsource lines makes the test fail.
So #731853 isn't really fixed, yet.
Actually, that's related to the problem I reported, as the underlying
issue for both is the same:
In the FindSrc function apt chooses a new 'best hit', if either
* there is a target release and it matches the release of the package,
* or the version of the package is higher than the last best hit.
Consider having 1.0 (stable), 2.0 (unstable) and 1.5 (unstable),
in this order.
Looking for the version in stable, apt first selects 1.0, because the
release matches the target release, but then subsequently selects 2.0,
because the version is higher.
Looking for the version in unstable, apt first selects 2.0, because the
release matches the target release, but then subsequently selects 1.5,
because the release also matches the target release.
The correct way would be to choose a new 'best hit', if either
* there is a target release and it matches the release of the package,
* or there is no target release
and the version is higher than the last best hit.
disable privilege dropping for rsh/ssh method by default
ssh expects various configuration bits to be usable like known hosts,
possibly keys and co. Setting this up needs some user work for probably
not a whole lot of benefits, so instead of forcing it upon users on
upgrade disable dropping for it by default.
disable privilege-drop verification by default as fakeroot trips over it
Dropping privileges is an involved process for code and system alike so
ideally we want to verify that all the work wasn't in vain. Stuff
designed to sidestep the usual privilege checks like fakeroot (and its
many alternatives) have their problem with this through, partly through
missing wrapping (#806521), partly as e.g. regaining root from an
unprivileged user is in their design. This commit therefore disables
most of these checks by default so that apt runs fine again in a
fakeroot environment.
This also deals with the unlikely case of groups being mentioned
multiple times or if the effective group isn't mentioned at all.
In practice, it is a debugging aid through like for #806475.
Michael Vogt [Fri, 27 Nov 2015 11:29:22 +0000 (12:29 +0100)]
Check if the Apt::Sandbox::User exists in CheckDropPrivsMustBeDisabled()
If it does not exist disabled priv dropping as there is nothing
we can drop to. This will unblock people with special chroots
or systems that deleted the "_apt" user.
Michael Vogt [Fri, 27 Nov 2015 11:07:48 +0000 (12:07 +0100)]
Deal with killed acquire methods properly instead of hanging
This fixes a regression caussed by commit 95278287f4e1eeaf5d96749d6fc9bfc53fb400d0
that moved the error detection of RunFds() later into the loop.
However this broke detecting issues like dead acquire methods.
Instead of relying on the global error state (which is bad)
we now pass a boolean value back from RunFds() and break on
false.
update manpage last-modified from git via pre-export hook
Never updating this information is wrong, updating it automatically
isn't super correct either, but it seems conventional to have it and
updating it more often than needed seems better than updating it never.
In 12f7536 I chose to opt for a slightly better text which avoids
assigning copyright to the FSF (not because it would be wrong to do it,
but is usually not what the contributor intended, but just the default.
xgettext has a --foreign flag for avoiding the copyright, but po4a
hasn't and an empty copyright-holder doesn't work either, so this little
rework of files with sed and cat.
show potentially arch-qualified fullname in 'apt show'
We do not show the architecture as a dedicated field as this is rather
technical information, but as packagename it makes sense to show the
architecture as other part of apt will refer to it in this way.
do not sent Last-Modified if we expect a changed file
In 8d041b4f we made apt figure out based on the last Release file it has
if it should request a file or not given that the hashes changed or not.
So if we have a last Release file and do a request, do not sent a
Last-Modified header as we expect a change so much that a non-change
would indeed be an error. The Last-Modified header is therefore at best
ignored by the server, so sending it is just wasted effort. In the worst
case as time is a fragile thing the server decides against sending us an
update with the idea that we already have the latest content, which we
know for a fact that we haven't. Given that we sent less information to
the server our request is on its own also less identifiable as coming
from a returning or new user.
The disadvantage is that if we end up getting an old index file after
getting a new Release file from another mirror the old mirror will not
be able to tell us 'Hit', but instead sends us the complete file we
discard, but both lets us end up with the same error class in the end,
so the difference isn't big in practice.
do not segfault in cache generation on mmap failure
Out of memory and similar circumstanzas could cause MMap::Map to fail
and especially the mmap/malloc calls in it. With some additional
checking we can avoid segfaults and similar in such situations – at
least in theory as if this is a real out of memory everything we do to
handle the error could just as well run into a memory problem as well…
But at least in theory (if MMap::Map is made to fail always) we can deal
with it so good that a user actually never sees a failure (as the cache
it tries to load with it fails and is discarded, so that DynamicMMap
takes over and a new one is build) instead of segfaulting.
do not rerun ./configure causing FTCBFS with newer autotools-dev
If the config.{sub,guess} files we linked in were newer than our
configure script we ended up recreating configure and then rerun it
without all the configuration options which were (potentially) present
for a previous run.
We avoid this by changing to the same ruleset as in the debian/rules
file which compares the config.* files against a stamp file rather than
the configure script itself as its the configuration itself which
depends on all scripts, not configure on the config scripts.
While at it, we also drop the 'make -s dirs' call as we don't need to do
it explicitly here as proper dependencies will take care of it.
Thanks: Helmut Grohne for the detailed bugreport. Closes: 804923
The hack introduced in aa91826f is replaced with a hopefully better
working "proper" solution with a new variable just for the standard we
use everywhere we use CXXFLAGS.
In ce1f3a2c we started warning about failing unlinking, which we
consistently do for directories. That isn't a problem as directories
usually aren't in the places we do want to clean up – with the potential
exeception of "lost+found", so lets ignore it like we ignore our own
partial/ subdirectory.
support setting empty values (sanely) & removing support for
space-gapping: '-o option= value'
That is a very old feature (straight from 1998), but it is super
surprising if you try setting empty values and instead get error
messages or a non-empty value as the next parameter is treated as the
value – which could have been empty, so if for some reason you need a
compatible way of setting an empty value try: '-o option="" ""'.
I can only guess that the idea was to support '-o option value', but we
survived 17 years without it, we will do fine in the future I guess.
Similar is the case for '-t= testing' even through '-t testing' existed
before and the code even tried to detect mistakes like '-t= -b' … all
gone now.
Technically that is as its removing a feature replacing it with another
a major interface break. In practice I really hope for my and their
sanity that nobody was using this; but if for some reaon you do: Remove
the space and be done.
I found the patch and the bugreport actually only after the fact, but
its reassuring that others are puzzled by this as well and hence a
thanks is in perfect order here as the patch is practical identical
[expect that this one here adds tests and other bonus items].
Thanks: Daniel Hartwig for initial patch. Closes: 693092
do not use _apt for file/copy sources if it isn't world-accessible
In 0940230d we started dropping privileges for file (and a bit later for
copy, too) with the intend of uniforming this for all methods. The
commit message says that the source will likely fail based on the
compressors already – and there isn't much secret in the repository
content. After all, after apt has run the update everyone can access the
content via apt anyway…
There are sources through which worked before which are mostly
single-deb (and those with the uncompressed files available).
The first one being especially surprising for users maybe, so instead of
failing, we make it so that apt detects that it can't access a source as
_apt and if so doesn't drop (for all sources!) privileges – but we limit
this to file/copy, so the uncompress which might be needed will still
fail – but that failed before this regression.
We display a notice about this, mostly so that if it still fails (e.g.
compressed) the user has some idea what is wrong.
Notices are just hints, but if they are printed in tests, they should be
expected and if not the test should fail. No current test has this
problem, so that is just potential future proving.
"support" unsigned Release files without hashes again
This 'ignores' the component Release files you can find in Debian
alongside the binary-* directories, which isn't exactly a common
usecase, but it worked before, so lets support it again as this isn't
worse than a valid Release file which is unsigned.
allow acquire method specific options via Binary scope
Allows users who know what they are getting themselves into with this
trick to e.g. disable privilege dropping for e.g. file:// until they can
fix up the permissions on those repositories. It helps also the test
framework and people with a similar setup (= me) to run in less modified
environments.
drop privileges in copy:// method as we do for file://
Continueing on the track of dropping privileges in all methods, lets
drop it in copy, too, as the reasoning for it is very similar to file
and the interaction between the too quiet interesting as copy kinda
surfed as a fallback for file not being able to read the file. Both now
show a better error message as well as it was previously claiming to
have a hashsum mismatch, given that it couldn't read the file.
allow getaddrinfo flag AI_ADDRCONFIG to be disabled
This flags is generally handy to avoid having to deal with ipv6 results on an
ipv4-only system, but it prevents e.g. the testcases from working if the
testsystem has no configured address at the moment (expect loopback), so
allow it to be sidestepped and let the testcases sidestep it.
Unlinking /dev/null is bad, we shouldn't do that. Also, we should print
at least a warning if we tried to unlink a file but didn't manage to
pull it of (ignoring the case were the file is /dev/null or doesn't
exist in the first place).
This got triggered by a relatively unlikely to cause problem in
pkgAcquire::Worker::PrepareFiles which would while temporary
uncompressed files (which are set to keep compressed) figure out that to
files are the same and prepare for sharing by deleting them. Bad move.
That also shows why not printing a warning is a bad idea as this hide
the error for in non-root test runs.
ensure FileFd doesn't try to open /dev/null as atomic and co
The wrapping will fail in the best case and actually end up deleting
/dev/null in the worst case. Given that there is no point in trying to
write atomically to /dev/null as you can't read from it again just
ignore these flags if higher level code ends up trying to use them on
/dev/null.
ignore newlines in dpkg-deb control output for installing debs
Leading or trailing newlines can be confusing for our parser as it
expects two newlines to start/stop a new stanza. To solve this the lines
we wanna add are printed first, ignore any leading newlines and then add
the stanza as provided by dpkg-deb with or without trailing newlines as
the parser will look at the first stanza only anyway and removing
trailing newlines is considerably harder to do.
support arch:all data e.g. in separate Packages file
Based on a discussion with Niels Thykier who asked for Contents-all this
implements apt trying for all architecture dependent files to get a file
for the architecture all, which is treated internally now as an official
architecture which is always around (like native). This way arch:all
data can be shared instead of duplicated for each architecture requiring
the user to download the same information again and again.
There is one problem however: In Debian there is already a binary-all/
Packages file, but the binary-any files still include arch:all packages,
so that downloading this file now would be a waste of time, bandwidth
and diskspace. We therefore need a way to decide if it makes sense to
download the all file for Packages in Debian or not. The obvious answer
would be a special flag in the Release file indicating this, which would
need to default to 'no' and every reasonable repository would override
it to 'yes' in a few years time, but the flag would be there "forever".
Looking closer at a Release file we see the field "Architectures", which
doesn't include 'all' at the moment. With the idea outlined above that
'all' is a "proper" architecture now, we interpret this field as being
authoritative in declaring which architectures are supported by this
repository. If it says 'all', apt will try to get all, if not it will be
skipped. This gives us another interesting feature: If I configure a
source to download armel and mips, but it declares it supports only
armel apt will now print a notice saying as much. Previously this was a
very cryptic failure. If on the other hand the repository supports mips,
too, but for some reason doesn't ship mips packages at the moment, this
'missing' file is silently ignored (= that is the same as the repository
including an empty file).
The Architectures field isn't mandatory through, so if it isn't there,
we assume that every architecture is supported by this repository, which
skips the arch:all if not listed in the release file.
projects / apt.git / log
