Dominic Benson [Mon, 20 Jun 2016 12:47:46 +0000 (13:47 +0100)]
Check for cached hash entries to determine which (if any) hash types
need to be generated for the current file.
In 1.0.9, each hash type was handled by a separate method, each of
which checked the cache. It looks like when these code paths were
unified (in a311fb96b84757ef8628e6a754232614a53b7891) the cache
checks were not incorporated into the new method.
implement and document DIRECT for auto-detect-proxy
There is a subtile difference between an empty setting and "DIRECT" in
the configuration as the later overrides the generic settings while the
earlier does not. Also, non-zero exitcodes should really be reported as
an error rather than silently discarded.
Commands are probably better of always having output through as the
fall through to the generic proxy settings is likely not intended. As
documenting and implementing this more consistently is kind of a
regression through, it is split off into the next commit.
avoid std::get_time usage to sidestep libstdc++6 bug
As reported upstream in
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71556
the implementation of std::get_time is currently not as accepting as
strptime is, especially in how hours should be formatted.
Just reverting 9febc2b238e1e322dce1f94ecbed46d595893b52 would be
possible, but then we would reopen the problems fixed by it, so instead
I opted here for a rewrite of the parsing logic which makes this method
a lot longer, but at least it provides the same benefits as the rewrite
in std::get_time was intended to give us and decouples us from the fix
of the issue in the standard library implementation of GCC.
merge sources.list lines based on Release filename
Merging by URI means that having sources lines with different URI
methods results in 'strange' warning and error messages, which aren't
very friendly from a user point of view as not encoding the method in
the filename is effectivly an implementation detail.
Merging by filename removes these messages and makes everything "work"
even if it isn't working the way it is configured as the indexes aren't
acquired over the method given, but over the first method for this
release file (which argueably is an implementation detail stemming from
the filename encoding, too).
So either direction isn't perfectly "right", but personally I prefer
"magic" over strange error messages (and doing a full-circle detection
of this with its own messages which would need to be translated feels
like way too much effort for dubious gain).
http: don't hang on redirect with length + connection close
Most servers who close the connection do not send a content-length as
this is redundant information usually, but some might and while testing
with our server and with 'aptwebserver::response-header::Connection' set
to 'close' I noticed that http hangs after a redirect in such cases, so
if we have the information, just use it instead of discarding it.
We usually use absolute paths to specific the location of dpkg, apt-key
and the like, but there is nothing wrong with using just the command
name and instead let exec(3) make the lookup in PATH.
We had a wild mixture before, so opting for the more accepting option
out of the two seems about right especially as it makes no difference in
the default case as apt uses absolute paths.
We had an old FIXME saying that it is probably pointless to do this if
we limit by length of the commandline already and I completely agree.
The splitting is bad enough if it must be done, so we should only do it
if we have to (as in absolute length of commandline) and, but that is
just a remark, it is unlikely that we ever have/had a call triggering
this as the default value was ~32000 items…
don't explicitly configure the last round of packages
We end our operation by calling "dpkg --configure -a", so instead of
running a (big) configure run with all packages mentioned explicitly
before this, we simply skip them and let them be handled by this call
implicitly.
There isn't really an observeable gain to be had here from a speed
point, but it helps in avoiding an (uncommon) problem of having a too
long commandline passed to dpkg, which we would split up (probably
incorrectly).
Most (if not all) solvers should be able to run perfectly fine without
root privileges as they get the entire state they are supposed to work
on via stdin and do not perform any action directly, but just pass
suggestions on via stdout.
The new default is to run them all as _apt hence, but each solver can
configure another user if it chooses/must. The security benefits are
minimal at best, but it helps preventing silly mistakes (see 35f3ed061f10a25a3fb28bc988fddbb976344c4d) and that is always good.
Note that our 'apt' and 'dump' solver already dropped privileges if they
had them.
It wasn't noticeable before, but now with the (optional) logging it can
be observed that we decide in the internal path two times if an internal
or external solver should be used (and hence with logging, it is
attempted twice), so if we are in the internal path call the internal
resolver directly, which means those internal methods need to be public
– but we can hide them based on the symbol at least.
edsp: optionally store a compressed copy of the last scenario
For bugreports and co it could be handy to have the scenario and all the
settings used in it around later for inspection for EDSP like protocols.
EDSP might not be the most interesting as the user can still interrupt
the process before the solution is applied and users tend to have an
opinion on the "rightness" of a solution, so it is disabled by default.
EDSP(-like) protocols are one-shot processes working on data which
exists only as long as they run (as they get feed via a pipe), so trying
to write a cache for it is pretty pointless, especially as it will
usually fail as the cache files tend to be owned by root, but the
process is run as a unpriviledged user (either _apt if called by root,
otherwise the user of the caller).
So this was in fact only observeable with our testcases which run as
non-root and the worst which happens is that a valid cache is overridden
with an invalid one which the next run will detect and not use.
With have better ways to compare, manipulate and work with strings, so
use it instead of counting string length by hand with is a wonder it
hasn't failed yet. Ignoreable from a changelog perspective as there is
no behaviour change.
The classes are all marked as hidden, so changing them is no problem ABI
wise and will help with introducing protocols similar to EDSP.
The change has no observeable behavior difference, its just code
juggling.
The script takes the version from the changelog, but if it lacks behind
and the symbols file already includes symbols tagged for the next
version the helper prints incorrect lines as NEW for these symbols, but
ideally it shouldn't print them at all as the symbol is already dealt
with.
edsp: use a stanza based interface for solution writing
EDSP had a WriteSolution method to write out the entire solution based
on the inspection of a given pkgDepCache, but that is rather inflexible
both for EDSP itself and for other EDSP like-protocols. It seems better
to use a smaller scope in printing just a single stanza based on a given
version as there is more reuse potential.
Currently an EDSP solver gets send basically all versions which means
the absolute count is the same, but that might not be true forever (and
with the skipping of rc-only versions it kinda is already) and even if
it were true, segfaulting on bad input seems wrong.
apt-key: change to / before find to satisfy its CWD needs
First seen on hurd, but easily reproducible on all systems by removing
the 'execution' bit from the current working directory and watching some
tests (mostly the no-output expecting tests) fail due to find printing:
"find: Failed to restore initial working directory: …"
Samuel Thibault says in the bugreport:
| To do its work, find first records the $PWD, then goes to
| /etc/apt/trusted.gpg.d/ to find the files, and then goes back to $PWD.
|
| On Linux, getting $PWD from the 700 directory happens to work by luck
| (POSIX says that getcwd can return [EACCES]: Search permission was denied
| for the current directory, or read or search permission was denied for a
| directory above the current directory in the file hierarchy). And going
| back to $PWD fails, and thus find returns 1, but at least it emitted its
| output.
|
| On Hurd, getting $PWD from the 700 directory fails, and find thus aborts
| immediately, without emitting any output, and thus no keyring is found.
|
| So, to summarize, the issue is that since apt-get update runs find as a
| non-root user, running it from a 700 directory breaks find.
Solved as suggested by changing to '/' before running find, with some
paranoia extra care taking to ensure the paths we give to find are really
absolute paths first (they really should, but TMPDIR=. or a similar
Dir::Etc::trustedparts setting could exist somewhere in the wild).
The commit takes also the opportunity to make these lines slightly less
error ignoring and the two find calls using (mostly) the same parameters.
Thanks: Samuel Thibault for 'finding' the culprit! Closes: 826043
ignore std::locale exeception on non-existent "" locale
In 8b79c94af7f7cf2e5e5342294bc6e5a908cacabf changing to usage of C++ way
of setting the locale causes us to be terminated in case of usage of an
ungenerated locale as LC_ALL (or similar) – but we don't want to fail
here, we just want to carry on as before with setlocale which we call in
that case just for good measure.
This reduces the number of symbols by about 10%. Unfortunately,
it does not seem to cover all the weird std::vector and friend
template expansions.
ABI should not brake due to that change: It was never specified
before whether an inline symbol was exported or not; so no library
could rely on its presence. Instead, the symbols were exported in
each library/program needing it and and then merged into a common
one by the dynamic linker.
Also update the symbol files to account for the removed symbols.
try to detect sudo spawned root-shell in prefixing
It is a try as the we need to inspect SUDO_COMMAND which could be
anything – apt, apt-get, in /usr/bin, in a $DPKG_ROOT "chroot", build
from source, aliases, …
The best we can do is look if the SHELL variable is equal to the
SUDO_COMMAND which would mean a shell was invoked. That isn't fail-safe
if different shells are involved as sub-shells have the tendency of not
overriding the SHELL so a bash started from within zsh can happily
pretend to be still zsh, so we could have a look at /etc/shells for a
list, but oh well, we have to stop somewhere I guess.
This sudo-prefixing feature is a gimmick after all.
The std::put_time and std::get_time introduced in 9febc2b238e1e322dce1f94ecbed46d595893b52 are part of C++11, but not
implemented in GCC until version 5. std::put_time could actually be
worked around via using the facets put() directly, but get() isn't
implemented so that doesn't really help.
We require various tools from wily (which also means we can't build apt
on Debian stable) already, so requiring gcc-5 is just one more instead
of a big step [and an ignoreable change for changelog anyhow].
It also helps in testing what will actually be used (in terms of the
c++11 std ABI) instead of the old ABI.
We use a wild mixture of C and C++ ways of generating output, so having
a consistent world-view in both styles sounds like a good idea and
should help in preventing regressions.
accept only the expected UTC timezones in date parsing
HTTP/1.1 hardcodes GMT (RFC 7231 §7.1.1.1) and what is good enough for the
internet must be good enough for us™ as we reuse the implementation
internally to parse (most) dates we encounter in various places like the
Release files with their Date and Valid-Until header fields.
Implementing a fully timezone aware parser just feels too hard for no
effective benefit as it would take 5+ years (= until LTS's are out of
fashion) until a repository could use non-UTC dates and expect it to
work. Not counting non-apt implementations which might or might not
only want to encounter UTC here as well.
As a bonus, this eliminates the use of an instance of setlocale in
libapt.
Setting the C++ locale via std::locale::global(std::locale("")); which
would otherwise default to the default C locale (aka: unaffected by
setlocale) effects the formatting of numeric types in IO streams, which
for output for humans is perfectly sensible, but breaks our many text
interfaces used and parsed by us and others without expecting the
numbers to be formatted.
libapt allows to configure compressors to be used by its system via
configuration implemented in 03bef78461c6f443187b60799402624326843396,
but that was never really documented and also only partly working, which
also explains why the tests weren't using it…
The debian/rules file tries to guess in which directory it is supposed
to be building, but that guess is always ./build – if it wasn't it
would fail later as not all rules take alternatives into acount.
So, as this is clearly not used lets remove this complexity instead of
fixing it up.
The code moving in eb1f04dda07c2b69549ad9fd793cca0e91841b3e
moved the acquire stuff above the simulation exit, so before getting
locks (and creating/chmod directories) we should be checking if we
should actually really do it…
avoid triggering gcc's -Wunsafe-loop-optimizations in EDSP
apt/apt-pkg/edsp.cc: In function ‘bool EDSP::WriteLimitedScenario(pkgDepCache&, FILE*, const PackageSet&, OpProgress*)’:
apt/apt-pkg/edsp.cc:245:56: warning: cannot optimize loop, the loop counter may overflow [-Wunsafe-loop-optimizations]
std::string dependencies[pkgCache::Dep::Enhances + 1];
^
Using a std::array to silence gcc as well as as a code improvement feels right here.
edsp: try harder to not generate unneeded error messages
The &= introduced in the EDSP-FileFd conversion isn't working to full
satisfaction for multiple && clauses as the && has a higher binding than
&= has, so that the methods were called even through they shouldn't
have because of previous errors. Using variadic functions we can solve
this in a slightly cleaner way bringing down the amount of 'broken pipe'
errors for the error case of the dump resolver substantially.
fail instead of segfault on unreadable config files
The report mentions "apt list --upgradable", but there are others which
have inconsistent behavior ranging from segfaulting to doing something
with the partial (and hence incomplete) data. We had a recent report
about sources.list (#818628), this one mentions prefences, the obvious
next step is conf files… so the testcase is adapted to check for all
three in file and directory versions and run a bunch of commands each
time which should all have more or less the same behavior in such a case
(aka error out).
--show-upgraded is the default since 906fbf8886926eeb302332d997c9bd861291e155 so documenting it as if it
would be an option having an effect as is feels wrong and we do the same
for other options like install-recomends, download, …, too.
This commit also removes -u from the documentation, but still supports
it in the commandline parsing. Eventually we should deprecate the short
option, but for now lets just stop documenting it.
show final solution in --no-download --fix-missing mode
This commit moves the creation of the fetcher and with it the
calculation of the filenames before the code generation the various
lists detailing the solution. This means that simulation comes even so
slightly closer to a real run as it will require and parse the package
indexes for filenames and queuing of URIs, so that a simulation "using"
an unavailable download method actually fails now.
The real benefit of this change is through that the rather special but
nontheless handy --no-download --fix-missing mode now actually shows
what the solution is it will apply to the system rather than the
solution it would if it could download all not-downloaded packages.
Errors cause a kind of automatic no already, but warnings and notices
are only displayed at the end of the apt execution even through they
could effect the choice of saying yes/no to questions: E.g. if a
configuration (file) was ignored you wanted to have an effect or if an
external solver you used generated warnings suggesting that the solution
might be valid, but bogus non-the-less and similar things.
Note that this only moves those messages up to the question if the
answer is interactive – not if e.g. -y is used or no question is asked at
all so this has an effect only on interactive usage of apt(-get), not
script who might be parsing apt output.
This fixes Debian/apt#13 and the launchpad bug listed below,
but is far more advanced. I went through private-cmndline.cc
and looked at the supported options.
LP: #1573547
Thanks: Elias Fröhner and Svyatoslav Gryaznov for the initial work
edsp: warn if unexpected stanzas appear in the solution
Unexpected are for examples removal requests for versions which aren't
installed, installations of already installed versions & requests to
install and remove a package at the same time.
Document that package identifiers must be unique (apt only uses the last
action for a given identifier) and that install requests do also imply
upgrades and downgrades (and thus removal of the old version). This is
to prevent that solvers express an upgrade or downgrade instruction as
two stanzas: a removal of the old version and an installation of the new
version. Instead, a single install stanza is sufficient to express
upgrade or downgrade requests.
The spec was slightly inconsistent if the preferences setting is
available only as generic or specific setting & the code only supported
the specific one, while for the strict-pinning was only generic…
As the usual pattern for apt is to have both options we adapt the spec
and code to support both as well.
This also adds a purely informal "Solver" field so in case the request
is saved in a file, we know to which solver the sent preferences apply.
update: Run Post-Invoke-Success if not all sources failed
Failures can happen and APT regardless will do a partial cache
update anyway. Because APT ensures that the list directory is
in a sane state, it makes sense to also call success hooks if
success was only partial - otherwise it loses sync with APT.
Most importantly, this causes the appstream cache to be empty,
see launchpad bug #1562733.
This is somewhat overly optimistic though: As soon as any repository
has nonexisting optional files, the missing optional files are also
treated as success, which means a single broken repository without an
InRelease file still runs Success hooks, even though it really should
not.
don't sent uninstallable rc-only versions via EDSP
Versions which are only available in dpkg/status aren't installable and
apt doesn't pick them as candidate for this reason – for the same reason
such packages shouldn't be sent to an external solver via EDSP. The
packages are pinned to -1, but if the solver has strict pinning disabled
it could end up picking this version anyhow – which is a request apt can
not satisfy.
Reported-By: Maximiliano Curia <maxy@debian.org> on IRC
gpg doesn't give use a UID on NODATA, which we were "expecting" (but not
using for anything), but just an error number. Instead of collecting
these as badsigners which will trigger a "invald signature" error with
remarks like "NODATA 1" we instead adapt a message similar to the NODATA
error of a clearsigned file (which is actually not reached anymore as we
split them up, which fails with a NOSPLIT error, which uses the same
general error message).
In other words: Not a security relevant change, just a user experience
improvement as we now point them to the most likely cause of the
problem instead of saying "invalid signature" which would point them in
the direction of the archive being broken (for everyone) instead.
A frontend like apt-file is only interested in a specific set of files
and selects those easily via "Created-By". If it supports two locations
for those files through it would need to select both and a user would
need to know that implementation detail for sources.list configuration.
The "Identifier" field is hence introduced which by default has the same
value as "Created-By", but can be freely configured – especially it can
be used to give two indexes the same identifier.
Sometimes index files are in different locations in a repository as it
is currently the case for Contents files which are per-component in
Debian, but aren't in Ubuntu. This has historic reasons and is perhaps
changed soon, but such cases of transitions can always happen in the
future again, so we should prepare:
Introduced is a new field declaring that the current item should only be
downloaded if the mentioned item wasn't allowing for transitions without
a flagday in clients and archives.
This isn't implemented 'simpler' with multiple MetaKeys as items (could)
change their descriptions and perhaps also other configuration bits with
their location.
download arch:all also for NATIVE_ARCHITECTURE indextargets
It looks a bit strange on the outside to have multiple "native
architecture", but all is considered an implementation detail and e.g.
packages of arch:all are in dependency resolution equal to native
packages.
don't construct MetaIndex acquire items with IndexTargets
We don't have to initialize the Release files with a set of IndexTargets
to acquire, but instead wait for the Release file to be acquired and
only then ask which IndexTargets to get.
Progress reporting used an "upper bound" on files we might get, expect
that this wasn't correct in case pdiff entered the picture. So instead
of calculating a value which is perhaps incorrect, we just accept that
we can't tell how many files we are going to download and just keep at
0% until we know. Additionally, if we have pdiffs we wait until we got
these (sub)index files, too.
That could all be done better by downloading all Release files first and
planing with them in hand accordingly, but one step at a time.
The code naturally evolved from a TransactionManager optional to a
required setup which resulted in various places doing unneeded checks
suggesting a more complicated setup than is actually needed.
fix same-mirror redirection for Release{,.gpg} pair
Commit 9b8034a9fd40b4d05075fda719e61f6eb4c45678 just deals with
InRelease properly and generates broken URIs in case the mirror (or the
achieve really) has no InRelease file.
[As this was in no released version no need to clutter changelog with a
fix notice.]
tests: disable generation of Release.gpg by default
Most tests just need a signed repository and don't care if it signed by
an InRelease file or a Release.gpg file, so we can save some time by
just generating one of them by default.
Sounds like not much, but quickly adds up to a few seconds with the
amount of tests we have accumulated by now.
allow redirection for items without a space in the desc again
Broken in a4b8112b19763cbd2c12b81d55bc7d43a591d610.
If an item has a description which includes no space and is redirected
to another mirror the code which wants to rewrite the description
expects a space in there, but can't find it and the unguarded substr
command on the string will fail with an exception thrown…
dpkg can optionally colorize its output since 1.18.5. Currently this
defaults to 'never', but it will eventually be 'auto'. It seems
reasonable to assume that a user who has enabled/disabled colors in apt
will want to have dpkg have the same state regarding color usage.
This isn't overriding explicit settings by the user, so in case a user
feels strongly about it one way or the other there are options.
The actual reason for this commit isn't the limit – there isn't much
point in using that much nesting – its in shutting up gcc mostly:
apt/apt-pkg/contrib/configuration.cc: In function ‘bool ReadConfigFile(Configuration&, const string&, const bool&, const unsigned int&)’:
apt/apt-pkg/contrib/configuration.cc:686:20: warning: cannot optimize loop, the loop counter may overflow [-Wunsafe-loop-optimizations]
string Stack[100];
^
by replacing this with C++s handy std::stack container (adapter).
Also cleans some whitespace noise from the file in the process.
warn if apt-key is run unconditionally in maintainerscript
We want to stop hard-depending on gnupg and for this it is essential
that apt-key isn't used in any critical execution path, which
maintainerscript are. Especially as it is likely that these script call
apt-key either only for (potentially now outdated cleanup) or still not
use the much simpler trusted.gpg.d infrastructure.
apt doesn't need gnupg in its main execution paths to function,
especially the Release file verification is done with gpgv only.
It is only used by apt-key for advanced key management functionality
most user will never use nor need.
The intend is to demote it eventually to Suggests, but we opt here for a
staged downgrade as there are still third-party repositories out there
which require apt-key functionality without depending on gnupg (or apt
for that matter).
projects / apt.git / log
