The bugreport shows a segfault caused by the code not doing the correct
magical dance to remove an item from inside a queue in all cases. We
could try hard to fix this, but it is actually better and also easier to
perform these checks (which cause instant failure) earlier so that they
haven't entered queue(s) yet, which in return makes cleanup trivial.
The result is that we actually end up failing "too early" as if we
wouldn't be careful download errors would be logged before that process
was even started. Not a problem for the acquire system, but likely to
confuse users and programs alike if they see the download process
producing errors before apt was technically allowed to do an acquire
(it didn't, so no violation, but it looks like it to the untrained eye).
Previously, we would have generated all the translations, but not
turn them on in the code. Instead, move the Translation crap into
po/ and disable po/ alltogether if USE_NLS if OFF.
This module should cover all sorts of large file supports, as long
as they either support the getconf LFS_CFLAGS command; or the
_FILE_OFFSET_BITS=64 or _LARGE_FILES macros.
do dpkg --configure before --remove/--purge --pending
Commit 7ec343309b7bc6001b465c870609b3c570026149 got us most of the way,
but the last mile was botched by having the pending calls in the wrong
order as this way we potentially 'force' dpkg to remove/purge a package
it doesn't want to as another package still depends on it and the
replacement isn't fully installed yet.
So what we do now is a configure before remove and purge (all with
--no-triggers) and finishing off with another configure pending call to
take care of the triggers.
Note that in the bugreport example our current planner is forcing dpkg
to remove the package earlier via --force-depends which we could do for
the pending calls as well and could be used as a workaround, but we want
to do less forcing eventually.
CMake: Translations: Build byproduct before output
This can lead to an inconsistent state otherwise, with the
output being updated and the byproduct not; for example,
when the build was manually interrupted.
debhelper 10 is much nicer with the installation part from
a dirty tree, so you can just fix some stuff breaking the
install step and then continue building with debuild -b -nc
until you have fixed all your stuff.
It also has some other advantages, of course, like some
bug fixes in shell escaping for maintscript, or systemd
helper changes.
debian: Make better use of the tree installed by CMake
This gets rid of the special casing of etc/apt, various
example file installations handled by the upstream build
system, and of course the directory creation for all dirs
created by the upstream build system.
tests/control: Handle the gpg1/gpg2 mess a bit better
Hardcoding gpgv1 and gnupg1 breaks Ubuntu, because on Ubuntu,
these packages do not exist yet. Instead allow gnupg (<< 2)
for gnupg1 and gnupg2 for gnupg (>= 2), so we cover all
potential combinations.
prepare-release: Use equivs and gdebi-core for travis deps
Our previous hack did not really support or groups and other
more advanced dependency types. This hack basically removes
build profiles, and the @-type depends for tests, and otherwise
converts the deps to a package which is then installed via
gdebi.
Instead of erroring out when receiving a SIGINT, let the
child deal with it - we'll error out anyway if the child
exits with an error or due to the signal. Also ignore
SIGQUIT, as system() ignores it.
This basically fixes Bug #832593, but: we are running
the hooks via sh -c. Some shells exit with a signal
error even if the command they are executing catches
the signal and exits successfully. So far, this has
been noticed on dash, which unfortunately, is our
default shell.
Example:
$ cat trap.sh
trap 'echo int' INT; sleep 10; exit 0
$ if dash -c ./trap.sh; then echo OK: $?; else echo FAIL: $?; fi
^Cint
FAIL: 130
$ if mksh -c ./trap.sh; then echo OK: $?; else echo FAIL: $?; fi
^Cint
OK: 0
$ if bash -c ./trap.sh; then echo OK: $?; else echo FAIL: $?; fi
^Cint
OK: 0
CMake: Translations: Don't rebuild .mo for line number changes
If only the line numbers changed in a file without any of the
translatable strings changing, the .po and .mo files were
rebuilt, making building simple code changes somewhat annoying.
We can work around this by passing --add-location=file to msgcomm
when we are creating the temporary .pot file used for building
the .mo files.
methods: read config in most to least specific order
The implementation of the generic config fallback did the fallback in
the wrong order so that the least specific option wasn't the last value
picked but in fact the first one… doh!
So in the bugreports case http -> https -> http::<hostname> ->
https::<hostname> while it should have been the reverse as before.
add --with-source option and Packages/Sources support
We support "./foobar.deb" as a way to install a deb file directly.
Recently .changes files were added. This highlights a problem as you
can't add the changes file without also trying to install all of them.
Now, it could also be handy to add entire Packages/Sources files to
perhaps get a bunch of packages in without installing them all
implicitly.
This commit introduces --with-source which allows to add *.deb, *.changes,
*.dsc, source-dirs, Packages & Sources files (the later can also be
compressed) without also installing them.
Fingerprints tend to be displayed in space-separated octet pairs so be
nice and allow delete to remove a key based on such a string rather than
requiring that the user is deleting all the spaces manually.
add the gpg-classic variant to the gpgv/gnupg or-group
We need to support partial upgrades anyhow, so we have to deal with the
different versions and your tests try to ensure that we do, so we
shouldn't make any explicit higher requirements.
gpg upstream committed "gpgv: Tweak default options for extra
security." applied on the 1.x and 2.x branches:
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=e32c575e0f3704e7563048eea6d26844bdfc494b
This commit includes "[…], but we should validate the key by its self
signature for primary key, and back signature for subkey."
Our testkeys are old and do not really considered best practices in the
last years, so their most recent self-signature is SHA1-only which with
this gpg commit and our testcases defaulting to --weak-digest sha1 are
refused.
So what we do here is just applying some of the recent best practices on
top of our testcase keys.
default to Dir=/ in dpkg/status file finding magic
Seen in cme #833656 if Dir isn't set (yet) we end up later absoluting a
path which was supposed to be absolute already, so if Dir is empty we
assume it to be '/' instead. In practice this is a bug in the software
using libapt, but for maxium compatibility lets explicitly set the
default value here to be safe.
support compression and by-hash for .diff/Index files
In af81ab9030229b4ce6cbe28f0f0831d4896fda01 by-hash got implemented as a
special compression type for our usual index files like Packages.
Missing in this scheme was the special .diff/Index index file containing
the info about individual patches for this index file. Deriving from the
index file class directly we inherent the compression handling
infrastructure and in this way also by-hash nearly for free.
support getting only-uncompressed files via by-hash
The URI we later want to modify to get the file via by-hash was unset in
case a file was only available uncompressed (which is usually not the
case) causing an acquire error.
In af81ab9030229b4ce6cbe28f0f0831d4896fda01 we implement by-hash as a
special compression type, which breaks this filesize setting as the code
is looking for a foobar.by-hash file then. Dealing this slightly gets
us the intended value. Note that this has no direct effect as this value
will be set in other ways, too, and could only effect progress reporting.
retry without same redirection mirror on 404 errors
If 9b8034a9fd40b4d05075fda719e61f6eb4c45678 serves the Release files
from a partial mirror we will end up getting 404 for some of the
indexes. Instead of giving up, we will instead ignore our same
redirection mirror constrain and ask the redirection service as a
potential hashsum mismatch is better than keeping the certain 404 error.
Now that we have the redirections loopchecker centrally in our items we
can use it also to prevent internal redirections to loop caused by
bugs as in a few instances we get into the business of rewriting the URI
we will query by ourself as we predict we would see such a redirect
anyway. Our code has no bugs of course, hence no practical difference. ;)
log with the failed item description, not with next try
The failure handling frequently changes URI & Description of the failed
item to try a slightly different combination which might work, but the
logging of the failure happens only afterwards as the same failure
handling decides if this is a critical error or not so we need a backup
here instead of potentially new content.
A purely cosmetic issue, but can still be confusing for humans.
If a server closes a connection after sending us a file that tends to
mean that its a type of server who always closes the connection – it is
therefore relatively pointless to try pipelining with it even if it
isn't a problem by itself: apt is just restarting the pipeline each
time after it got served one file and the connection is closed.
The problem starts if one or more proxies are between the server and apt
and they disagree about how the connection should be as in the
bugreporters case where the responses apt gets contain both Keep-Alive
and Proxy-Connection headers (which apt both ignores) indicating a
proxy is trying to keep a connection open while the response also
contains "Connection: close" indicating the opposite which apt
understands and respects as it is required to do.
We avoid stepping into this abyss by not performing pipelining anymore
if we got a respond with the indication to close connection if the
response was otherwise a success – error messages are sent by some
servers via this method as their pages tend to be created dynamically
and hence their size isn't known a priori to them.
don't sent Range requests if we know its not accepted
If the server told us in a previous request that it isn't supporting
Ranges with bytes via an Accept-Ranges header missing bytes, we don't
try to formulate requests using Ranges.
We keep various information bits about the server around, some only
effecting the currently handled file (like sizes) while others
should be persistent (like pipeline detections). http used to reset all
file-related manually, which is a bit silly if we already have a Reset()
method – which does reset all through –, so extending it with a
parameter for reuse and calling it from https too (as this was
previously resetting by just creating a new state struct – it uses no
value of the persistent state-keeping yet as it supports no pipelining).
Change anonscm.d.o links to /git/apt/apt.git and https
This also fixes Debian/apt#20, but is slightly more complete. I
think /git also looks better than /cgit, so let's switch the Vcs
entry in control over too.
It seems completely pointless from a server-POV to sent empty header
fields, so most of them don't do it (simply proven by this limitation
existing since day one) – but it is technically allowed by the RFC as
the surounding whitespaces are optional and Github seems to like sending
"X-Geo-Block-List:\r\n" since recently (bug reports in other http
clients indicate July) at least sometimes as the reporter claims to have
seen it on https only even through it can happen with both.
drop incorrect const attribute from DirectoryExists
Since its existence in 2010 DirectoryExists was always marked with this
attribute, but for no real reason. Arguably a check for the existence of
the file is not modifying global state, so theoretically this shouldn't
be a problem. It is wrong from a logical point of view through as
between two calls the directory could be created so the promise we made
to the compiler that it could remove the second call would be wrong, so
API wise it is wrong.
It's a bit mysterious that this is only observeable on ppc64el and can be
fixed by reordering code ever so slightly, but in the end its more our
fault for adding this attribute than the compilers fault for doing
something silly based on the attribute.
We use clock() as a very cheap way of getting a "random" value, but the
manpage warns that this could return -1, so we should be dealing with
this. Additionally, e.g. on hurd-i386 the value increases only slowly –
to slow for our fast running tests for randomness hence producing the
same range in both samples, so we introduce a simple busy-wait loop (as
clock is counting processor time used by the program) in the test which
delays the second sample just enough making our randomness a bit more
predictable.
Comparing floating numbers is always fun and in this instance a 9 < 9.0
is "somehow" true on hurd-i386 letting the tests fail by reporting that
too much progress achieved. A bit mysterious, but with some rework we
can use code which avoids dealing with the floats in this way entirely
and make our testcases happy.
ctest as run by cmake by default does not show the output of the tests
even if the tests failed. In terms of our tests it could be handy to set
it always, but unfortunately it seems like cmake doesn't allow it if the
internet is to be believed, so lets enable it at least while building
packages and on travis.
I don't know what happened back in 2009 when I wrote this,
but it seems I used the wrong option. These files should
not have any variable substitution done to them.
This commit looks heavy. Most of that comes from the fact that the
ordering of files in the translations changed with the switch to
CMake. I could have gone the extra mile to figure out the original
ordering and replicate it, but I have chosen to re-order everything
by file and line number, as that's easier.
With cmake using BUILDDIRECTORY at this place is not only as wrong as it
was before, but it might not even work always copying the system
provided one which might or might not be current and hence fails tests
needing it to be current like ./test-apt-move-and-forget-manual-sections
We don't want to always use the one from the source directory through
either like in autopkgtests.
http: auto-configure for local Tor proxy if called as 'tor'
With apts http transport supporting socks5h proxies and all the work
in terms of configuration of methods based on the name it is called with
it becomes surprisingly easy to implement Tor support equally (and
perhaps even a bit exceeding) what is available currently in
apt-transport-tor.
How this will turn out to be handled packaging wise we will see in
https://lists.debian.org/deity/2016/08/msg00012.html , but until this is
resolved we can add the needed support without actively enabling it for
now, so that this can be tested better.
block direct connections to .onion domains (RFC7687)
Doing a direct connect to an .onion address (if you don't happen to use
it as a local domain, which you shouldn't) is bound to fail and does
leak the information that you do use Tor and which hidden service you
wanted to connect to to a DNS server. Worse, if the DNS is poisoned and
actually resolves tricking a user into believing the setup would work
correctly…
This does block also the usage of wrappers like torsocks with apt, but
with native support available and advertised in the error message this
shouldn't really be an issue.
disable explicit configuration of all packages at the end
With b4450f1dd6bca537e60406b2383ab154a3e1485f we dropped what we
calculated here later on and now that we don't need it in the meantime
either we can just skip the busy work by default and expect dpkg to do
the right thing dropping also our little "last explicit configures"
removal trick introduced in b4450f1dd6bca537e60406b2383ab154a3e1485f.
This enables the last of a bunch of previously experimental options,
some of them existing still, but are very special and hence not really
worth documenting anymore (especially as it would need to be rewritten
now entirely) which is why the documentation is nearly completely
dropped.
The order of configuration stanzas in the simulation code changes
slightly as it isn't concerning itself with finding the 'right' order,
but any order is valid anyhow as long as the entire set happens in the
same call.
If a planner lets actions to be figured out by dpkg in pending calls
these actions aren't mentioned in a simulation. While that might be
a good thing for debugging, it would be a change in behavior and
especially if a planner avoids explicit removals could be confusing for
users. As such we perform the same 'trick' as in the dpkg implementation
by performing explicitly what would be done by the pending calls.
To save us some work and avoid desyncs we perform a layer violation by
using deb/ code in the generic simulation – and further we perform ugly
dynamic_cast to avoid breaking the ABI for nothing; aptitude is the only
other user of the simulation class according to codesearch.d.n and for
that our little trick works. It just isn't working if you happen to
extend pkgSimulate or otherwise manage to call the protected Go methods
directly – which isn't very realistic/practical.
The user has to approve the removal of a crossgraded package as it might
be needed to remove it (temporarily) in the process, but in most cases
we can happily avoid it and let dpkg unpack over it skipping the
remove. This has some effects on progress reporting and how deal with
selections through which makes this a tiny bit complicated.
allow methods to be disabled and redirected via config
To prevent accidents like adding http-sources while using tor+http it
can make sense to allow disabling methods. It might even make sense to
allow "redirections" and adding "symlinked" methods via configuration.
This could e.g. allow using different options for certain sources by
adding and configuring a "virtual" new method which picks up the config
based on the name it was called with like e.g. http does if called as
tor+http.
Socks support is a requested feature in sofar that the internet is
actually believing Acquire::socks::Proxy would exist. It doesn't and
this commit isn't adding it as that isn't how our configuration works,
but it allows Acquire::http::Proxy="socks5h://…". The HTTPS method was
changed already to support socks proxies (all versions) via curl. This
commit implements only SOCKS5 (RFC1928) with no auth or pass&user auth
(RFC1929), but not GSSAPI which is required by the RFC. The 'h' in the
protocol name further indicates that DNS resolution is delegated to the
socks proxy rather than performed locally.
The implementation works and was tested with Tor as socks proxy for
which implementing socks5h only can actually be considered a feature.
The https method implemented for a long while now a hardcoded fallback
to the same options in http, which, while it works, is rather inflexible
if we want to allow the methods to use another name to change their
behavior slightly, like apt-transport-tor does to https – most of the
diff being s#https#tor#g which then fails to do the full circle
fallthrough tor -> https -> http for https sources. With this config
infrastructure this could be implemented now.
use the same redirection handling for http and https
cURL which backs our https implementation can handle redirects on its
own, but by dealing with them on our own we gain finer control over which
redirections will be performed (we don't like https → http) and by whom
so that redirections to other hosts correctly spawn a new https method
dealing with these instead of letting the current one deal with it.
detect redirection loops in acquire instead of workers
Having the detection handled in specific (http) workers means that a
redirection loop over different hostnames isn't detected. Its also not a
good idea have this implement in each method independently even if it
would work
suggest transport-packages based on established namescheme
apt-transports not shipped in apt directly are usually named
apt-transport-% with % being what is in the name of the transport.
tor additional introduced aliases via %+something, which isn't a bad
idea, so be strip the +something part from the method name before
suggesting the installation of an apt-transport-% package.
This avoids us the maintainance of a list of existing transports
creating a two class system of known and unknown transports which would
be quite arbitrary and is unfriendly to backports.
ensure all configures are reported to hook scripts
A planner might not explicitly configure all packages, but we need to
know all packages which will be configured for progress reporting and to
tell the hook scripts about them as they rely on this for their own
functionality.
don't purge directly, but remove and do purge at the end
If we want a package to be purged from the system tell dpkg in the
ordering (if it has to touch it explicitly) to remove it and cover the
purging of the config files at the end with a --purge --pending call.
That should help packages move conffiles around between packages
correctly even if the user is purging packages directly in big actions
like dist-upgrades involving many packages.
Implemented a long while ago now with relatively good progress reporting
involving triggers is a good time to try delaying the execution of
triggers across dpkg invocations finally by default.
Note: The bugreport talks also about 'smarter' configuration which is a
much bigger part and approached from multiple directions, but doesn't
really involve triggers per-se so considering it decoupled should help
in getting it done…
Telling dpkg early on that we are going to remove these packages later
helps it with auto-deconfiguration decisions and its another area where
a planner can ignore the nitty gritty details and let dpkg decide the
course of action if there are no special requirements.
save and restore selection states before/after calling dpkg
dpkg decides certain things on its own based on selections and
especially if we want to call --pending on purge/remove actions, we need
to ensure a clean slate or otherwise we surprise the user by removing
packages we weren't allowed to remove by the user in this run (the
selection might be an overarching plan for the not-yet "future").
Ideally dpkg would have some kind of temporal selection interface for
this case, but it hasn't, so we make it temporal with the risk of
loosing state if we don't manage to restore them.
use dpkg --unpack --recursive to avoid long cmdlines
Having long commandlines split into two is a huge problem if it happens
and additionally if we want to introduce planners which perform less
micromanagment its a good idea to leave the details for dpkg to decide.
In practice this doesn't work yet unconditionally as a bug is hiding in
the ordering code of dpkg, but it works if apt imposes its ordering so
this commit allows for now at least to solve the first problem.
pass --force-remove-essential to dpkg only if needed
APT (usually) knows which package is essential or not, so we can avoid
passing this force flag to dpkg unconditionally if the user hasn't
chosen a non-default essential handling obscuring the information.
prepare-release: Switch over to CMake, set version in CMakeLists.txt
Teach the prepare-release script about the version new locations
and also set the version in CMakeLists, as that is better than
reading it from the changelog: CMake would not rerun automatically
otherwise if the version changed.
CMake: Rewrite existing Documentation support and add doxygen
This can now build all documentation. It should also be fairly
reusable for other projects, as long as they follow the same
naming scheme for the po4a output files and set the PACKAGE_*
variables used here.
We could have done all translations in a single call to po4a
like the makefile based buildsystem does. While that would
have made the output slightly nicer, this solution offers a
huge performance gain because it can translate the documents
in parallel, which also means that the xsltproc stage does not
have to wait for all translations to be done first.
You might think that the add_custom_command() should list the
actual output files as BYPRODUCTS. This is not true however:
Because the files are not always generated, Ninja will think
missing byproducts mean that the target is out of date - which
is not what we want.
Finally, also add the missing doxygen support. Note that the
packaging script cleans up some md5 and map files created by
doxygen, otherwise it is fairly boring.
This was dropped in autotools as I found no use of the HAVE_PTSNAME_R
macro. Turns out it was typoed as HAVE_PTS_NAME_R. Fix the #ifdef and
add checks to CMake for it.
Add support for our GTest based unit tests. By default, CMake will
look in /usr/src/gtest for the external GTest project, but this can
be overriden by defining GTEST_ROOT when invoking cmake.
projects / apt.git / log
