Without randomizing the order in which we download the index files we
leak needlessly information to the mirrors of which architecture is
native or foreign on this system. More importantly, we leak the order in
which description translations will be used which in most cases will e.g.
have the native tongue first.
Note that the leak effect in practice is limited as apt detects if a file
it wants to download is already available in the latest version from a
previous download and does not query the server in such cases. Combined
with the fact that Translation files are usually updated infrequently
and not all at the same time, so a mirror can never be sure if it got asked
about all files the user wants.
FreeBSD has two iconv systems: It ships an iconv.h itself,
and symbols for that in the libc. But there's also the port
of GNU libiconv, which unfortunately for us, Doxygen depends
on.
This changes things to prefer a separate libiconv library
over the system one; that is, the port on FreeBSD.
The host system might not have a dpkg installed, which makes
dpkg fail with:
dpkg not recorded as installed, cannot check for multi-arch support!
That's entirely useless of course. We want to know if dpkg could
support multi-arch in our chroot, so we pseudo-install dpkg into
the chroot and pretend it's version is one version higher than
the minimum dpkg version, so dpkg --assert-multi-arch works on
recent dpkgs.
test: Use printf "%b\n" instead of echo for strings with '\'
Use of echo with special characters is not portable. On a normal
POSIX system, the behavior with backslash escaped strings is
implementation-defined. On an XSI-conformant system, they must
be interpreted.
A way out is the printf command - printf "%b" specifies that
the following argument is to be printed with backslash escapes
interpreted.
Use /dev/fd in test-bug-712116-dpkg-pre-install-pkgs-hook-multiarch,
skip test-no-fds-leaked-to-maintainer-scripts (it is not guaranteed
that /dev/fd contains all file descriptors), and avoid the unneeded
use of /proc/fd in another test case.
This allows other vendors to use different paths, or to build
your own APT in /opt for testing. Note that this uses + 1 in
some places, as the paths we receive are absolute, but we need
to strip of the initial /.
Use C locale instead of C.UTF-8 for protocol strings
The C.UTF-8 locale is not portable, so we need to use C, otherwise
we crash on other systems. We can use std::locale::classic() for
that, which might also be a bit cheaper than using locale("C").
This basically just links everything to libintl if USE_NLS is
on. It would be better to just link those targets that are
actually translated, but doing so is a huge PITA.
Also move the include_directories() for the build-tree include/
directory to the top of the CMakeLists.txt, otherwise it only
gets passed after Intl_INCLUDE_DIRS, which means we will built
against installed apt-pkg headers (if any) instead of our own.
The BSD systems still ship their own db.h with a historical
BSD implementation, which is preferred by CMake, as it searches
its default path first. We thus have to disable the DEFAULT_PATH
for the search, unfortunately. We also need to pass the correct
include directory to the target.
Furthermore, on FreeBSD the library is called db-<VERSION>, so
let's add db-5 to the list of allowed names.
Ubuntu uses *.ddeb files for their debug packages, but the interface we
are using since f495992428a396e0f98886c9a761a804aa161c68 to talk to dpkg
isn't supporting *.ddeb files. This used to work previously as apt itself
isn't caring about the filenames at all and if they are explicitly
mentioned dpkg will accept all, too.
It might or might not be a good idea to patch dpkg, too, but regardless
of it happening, we don't want to couple us to closely to dpkg for this
minor feature but testing for this at runtime as it would delay shipping
the fix for the too long commandlines further.
It is also questionable if it is really a good idea to allow any file
extension to be used here (like .foobar in the testcase), but we used to
and we tend to avoid breaking existing usecases if we can help it.
As a bonus, this also allows the installation of ddeb files directly
from the commandline as you can with deb files already. We continue to
ignore udeb through as the user-mistake to useful ratio is too high.
In 105503b4b470c124bc0c271bd8a50e25ecbe9133 we got a warning implemented
for unreadable files which greatly improves the behavior of apt update
already as everything will work as long as we don't need the keys
included in these files. The behavior if they are needed is still
strange through as update will fail claiming missing keys and a manual
test (which the user will likely perform as root) will be successful.
Passing the new warning generated by apt-key through to apt is a bit
strange from an interface point of view, but basically duplicating the
warning code in multiple places doesn't feel right either. That means we
have no translation for the message through as apt-key has no i18n yet.
It also means that if the user has a bunch of sources each of them will
generate a warning for each unreadable file which could result in quite
a few duplicated warnings, but "too many" is better than none.
apt-key: warn instead of fail on unreadable keyrings
apt-key has inconsistent behaviour if it can't read a keyring file:
Commands like 'list' skipped silently over such keyrings while 'verify'
failed hard resulting in apt to report cconfusing gpg errors (#834973).
As a first step we teach apt-key to be more consistent here skipping in
all commands over unreadable keyrings, but issuing a warning in the
process, which is as usual for apt commands displayed at the end of the
run.
do not restore selections for already purged packages
In most cases apt was already skipping the (re)setting of packages as
to be removed/purged if dpkg had told us that it already did, but we
haven't dealt with it in the most obvious of the cases: Selections set
for packages we touched in this operation which either restores
selections even dpkg would have overridden or e.g. tries to restore a
purge selection for a package which was just purged – does not happen
with apt itself as it isn't using selections in this way, but higher
frontends like aptitude do.
The result in the later case is a warning printed by dpkg that we try to
set selections for an unknown package, which is harmless per se, but can
be confusing for users and we really shouldn't cause warnings in dpkg if
we can help it.
The bugreport shows a segfault caused by the code not doing the correct
magical dance to remove an item from inside a queue in all cases. We
could try hard to fix this, but it is actually better and also easier to
perform these checks (which cause instant failure) earlier so that they
haven't entered queue(s) yet, which in return makes cleanup trivial.
The result is that we actually end up failing "too early" as if we
wouldn't be careful download errors would be logged before that process
was even started. Not a problem for the acquire system, but likely to
confuse users and programs alike if they see the download process
producing errors before apt was technically allowed to do an acquire
(it didn't, so no violation, but it looks like it to the untrained eye).
Previously, we would have generated all the translations, but not
turn them on in the code. Instead, move the Translation crap into
po/ and disable po/ alltogether if USE_NLS if OFF.
This module should cover all sorts of large file supports, as long
as they either support the getconf LFS_CFLAGS command; or the
_FILE_OFFSET_BITS=64 or _LARGE_FILES macros.
do dpkg --configure before --remove/--purge --pending
Commit 7ec343309b7bc6001b465c870609b3c570026149 got us most of the way,
but the last mile was botched by having the pending calls in the wrong
order as this way we potentially 'force' dpkg to remove/purge a package
it doesn't want to as another package still depends on it and the
replacement isn't fully installed yet.
So what we do now is a configure before remove and purge (all with
--no-triggers) and finishing off with another configure pending call to
take care of the triggers.
Note that in the bugreport example our current planner is forcing dpkg
to remove the package earlier via --force-depends which we could do for
the pending calls as well and could be used as a workaround, but we want
to do less forcing eventually.
CMake: Translations: Build byproduct before output
This can lead to an inconsistent state otherwise, with the
output being updated and the byproduct not; for example,
when the build was manually interrupted.
debhelper 10 is much nicer with the installation part from
a dirty tree, so you can just fix some stuff breaking the
install step and then continue building with debuild -b -nc
until you have fixed all your stuff.
It also has some other advantages, of course, like some
bug fixes in shell escaping for maintscript, or systemd
helper changes.
debian: Make better use of the tree installed by CMake
This gets rid of the special casing of etc/apt, various
example file installations handled by the upstream build
system, and of course the directory creation for all dirs
created by the upstream build system.
tests/control: Handle the gpg1/gpg2 mess a bit better
Hardcoding gpgv1 and gnupg1 breaks Ubuntu, because on Ubuntu,
these packages do not exist yet. Instead allow gnupg (<< 2)
for gnupg1 and gnupg2 for gnupg (>= 2), so we cover all
potential combinations.
prepare-release: Use equivs and gdebi-core for travis deps
Our previous hack did not really support or groups and other
more advanced dependency types. This hack basically removes
build profiles, and the @-type depends for tests, and otherwise
converts the deps to a package which is then installed via
gdebi.
Instead of erroring out when receiving a SIGINT, let the
child deal with it - we'll error out anyway if the child
exits with an error or due to the signal. Also ignore
SIGQUIT, as system() ignores it.
This basically fixes Bug #832593, but: we are running
the hooks via sh -c. Some shells exit with a signal
error even if the command they are executing catches
the signal and exits successfully. So far, this has
been noticed on dash, which unfortunately, is our
default shell.
Example:
$ cat trap.sh
trap 'echo int' INT; sleep 10; exit 0
$ if dash -c ./trap.sh; then echo OK: $?; else echo FAIL: $?; fi
^Cint
FAIL: 130
$ if mksh -c ./trap.sh; then echo OK: $?; else echo FAIL: $?; fi
^Cint
OK: 0
$ if bash -c ./trap.sh; then echo OK: $?; else echo FAIL: $?; fi
^Cint
OK: 0
CMake: Translations: Don't rebuild .mo for line number changes
If only the line numbers changed in a file without any of the
translatable strings changing, the .po and .mo files were
rebuilt, making building simple code changes somewhat annoying.
We can work around this by passing --add-location=file to msgcomm
when we are creating the temporary .pot file used for building
the .mo files.
methods: read config in most to least specific order
The implementation of the generic config fallback did the fallback in
the wrong order so that the least specific option wasn't the last value
picked but in fact the first one… doh!
So in the bugreports case http -> https -> http::<hostname> ->
https::<hostname> while it should have been the reverse as before.
add --with-source option and Packages/Sources support
We support "./foobar.deb" as a way to install a deb file directly.
Recently .changes files were added. This highlights a problem as you
can't add the changes file without also trying to install all of them.
Now, it could also be handy to add entire Packages/Sources files to
perhaps get a bunch of packages in without installing them all
implicitly.
This commit introduces --with-source which allows to add *.deb, *.changes,
*.dsc, source-dirs, Packages & Sources files (the later can also be
compressed) without also installing them.
Fingerprints tend to be displayed in space-separated octet pairs so be
nice and allow delete to remove a key based on such a string rather than
requiring that the user is deleting all the spaces manually.
add the gpg-classic variant to the gpgv/gnupg or-group
We need to support partial upgrades anyhow, so we have to deal with the
different versions and your tests try to ensure that we do, so we
shouldn't make any explicit higher requirements.
gpg upstream committed "gpgv: Tweak default options for extra
security." applied on the 1.x and 2.x branches:
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=e32c575e0f3704e7563048eea6d26844bdfc494b
This commit includes "[…], but we should validate the key by its self
signature for primary key, and back signature for subkey."
Our testkeys are old and do not really considered best practices in the
last years, so their most recent self-signature is SHA1-only which with
this gpg commit and our testcases defaulting to --weak-digest sha1 are
refused.
So what we do here is just applying some of the recent best practices on
top of our testcase keys.
default to Dir=/ in dpkg/status file finding magic
Seen in cme #833656 if Dir isn't set (yet) we end up later absoluting a
path which was supposed to be absolute already, so if Dir is empty we
assume it to be '/' instead. In practice this is a bug in the software
using libapt, but for maxium compatibility lets explicitly set the
default value here to be safe.
support compression and by-hash for .diff/Index files
In af81ab9030229b4ce6cbe28f0f0831d4896fda01 by-hash got implemented as a
special compression type for our usual index files like Packages.
Missing in this scheme was the special .diff/Index index file containing
the info about individual patches for this index file. Deriving from the
index file class directly we inherent the compression handling
infrastructure and in this way also by-hash nearly for free.
support getting only-uncompressed files via by-hash
The URI we later want to modify to get the file via by-hash was unset in
case a file was only available uncompressed (which is usually not the
case) causing an acquire error.
In af81ab9030229b4ce6cbe28f0f0831d4896fda01 we implement by-hash as a
special compression type, which breaks this filesize setting as the code
is looking for a foobar.by-hash file then. Dealing this slightly gets
us the intended value. Note that this has no direct effect as this value
will be set in other ways, too, and could only effect progress reporting.
retry without same redirection mirror on 404 errors
If 9b8034a9fd40b4d05075fda719e61f6eb4c45678 serves the Release files
from a partial mirror we will end up getting 404 for some of the
indexes. Instead of giving up, we will instead ignore our same
redirection mirror constrain and ask the redirection service as a
potential hashsum mismatch is better than keeping the certain 404 error.
Now that we have the redirections loopchecker centrally in our items we
can use it also to prevent internal redirections to loop caused by
bugs as in a few instances we get into the business of rewriting the URI
we will query by ourself as we predict we would see such a redirect
anyway. Our code has no bugs of course, hence no practical difference. ;)
log with the failed item description, not with next try
The failure handling frequently changes URI & Description of the failed
item to try a slightly different combination which might work, but the
logging of the failure happens only afterwards as the same failure
handling decides if this is a critical error or not so we need a backup
here instead of potentially new content.
A purely cosmetic issue, but can still be confusing for humans.
If a server closes a connection after sending us a file that tends to
mean that its a type of server who always closes the connection – it is
therefore relatively pointless to try pipelining with it even if it
isn't a problem by itself: apt is just restarting the pipeline each
time after it got served one file and the connection is closed.
The problem starts if one or more proxies are between the server and apt
and they disagree about how the connection should be as in the
bugreporters case where the responses apt gets contain both Keep-Alive
and Proxy-Connection headers (which apt both ignores) indicating a
proxy is trying to keep a connection open while the response also
contains "Connection: close" indicating the opposite which apt
understands and respects as it is required to do.
We avoid stepping into this abyss by not performing pipelining anymore
if we got a respond with the indication to close connection if the
response was otherwise a success – error messages are sent by some
servers via this method as their pages tend to be created dynamically
and hence their size isn't known a priori to them.
don't sent Range requests if we know its not accepted
If the server told us in a previous request that it isn't supporting
Ranges with bytes via an Accept-Ranges header missing bytes, we don't
try to formulate requests using Ranges.
We keep various information bits about the server around, some only
effecting the currently handled file (like sizes) while others
should be persistent (like pipeline detections). http used to reset all
file-related manually, which is a bit silly if we already have a Reset()
method – which does reset all through –, so extending it with a
parameter for reuse and calling it from https too (as this was
previously resetting by just creating a new state struct – it uses no
value of the persistent state-keeping yet as it supports no pipelining).
Change anonscm.d.o links to /git/apt/apt.git and https
This also fixes Debian/apt#20, but is slightly more complete. I
think /git also looks better than /cgit, so let's switch the Vcs
entry in control over too.
It seems completely pointless from a server-POV to sent empty header
fields, so most of them don't do it (simply proven by this limitation
existing since day one) – but it is technically allowed by the RFC as
the surounding whitespaces are optional and Github seems to like sending
"X-Geo-Block-List:\r\n" since recently (bug reports in other http
clients indicate July) at least sometimes as the reporter claims to have
seen it on https only even through it can happen with both.
drop incorrect const attribute from DirectoryExists
Since its existence in 2010 DirectoryExists was always marked with this
attribute, but for no real reason. Arguably a check for the existence of
the file is not modifying global state, so theoretically this shouldn't
be a problem. It is wrong from a logical point of view through as
between two calls the directory could be created so the promise we made
to the compiler that it could remove the second call would be wrong, so
API wise it is wrong.
It's a bit mysterious that this is only observeable on ppc64el and can be
fixed by reordering code ever so slightly, but in the end its more our
fault for adding this attribute than the compilers fault for doing
something silly based on the attribute.
We use clock() as a very cheap way of getting a "random" value, but the
manpage warns that this could return -1, so we should be dealing with
this. Additionally, e.g. on hurd-i386 the value increases only slowly –
to slow for our fast running tests for randomness hence producing the
same range in both samples, so we introduce a simple busy-wait loop (as
clock is counting processor time used by the program) in the test which
delays the second sample just enough making our randomness a bit more
predictable.
Comparing floating numbers is always fun and in this instance a 9 < 9.0
is "somehow" true on hurd-i386 letting the tests fail by reporting that
too much progress achieved. A bit mysterious, but with some rework we
can use code which avoids dealing with the floats in this way entirely
and make our testcases happy.
ctest as run by cmake by default does not show the output of the tests
even if the tests failed. In terms of our tests it could be handy to set
it always, but unfortunately it seems like cmake doesn't allow it if the
internet is to be believed, so lets enable it at least while building
packages and on travis.
I don't know what happened back in 2009 when I wrote this,
but it seems I used the wrong option. These files should
not have any variable substitution done to them.
This commit looks heavy. Most of that comes from the fact that the
ordering of files in the translations changed with the switch to
CMake. I could have gone the extra mile to figure out the original
ordering and replicate it, but I have chosen to re-order everything
by file and line number, as that's easier.
projects / apt.git / log
