From: Michael Vogt Date: Fri, 22 May 2015 15:01:03 +0000 (+0200) Subject: Merge branch 'debian/sid' into debian/experimental X-Git-Tag: 1.1.exp10~5^2~4^2 X-Git-Url: https://git.saurik.com/apt.git/commitdiff_plain/4fc6b7570c3e97b65c118b58cdf6729fa94c9b03 Merge branch 'debian/sid' into debian/experimental Conflicts: apt-pkg/pkgcache.h debian/changelog methods/https.cc methods/server.cc test/integration/test-apt-download-progress --- 4fc6b7570c3e97b65c118b58cdf6729fa94c9b03 diff --cc debian/changelog index b0d518b17,00322b8c4..7414b5c61 --- a/debian/changelog +++ b/debian/changelog @@@ -1,239 -1,13 +1,249 @@@ +apt (1.1~exp8) experimental; urgency=medium + + [ Michael Vogt ] + * merge unstable upload version 1.0.9.3 + * Ensure /etc/apt/auth.conf has _apt:root owner + * Use sysconf(_SC_ARG_MAX) to find the size of Dpkg::MaxArgBytes + * Only support Translation-* that are listed in the {In,}Release file + * Call "Dequeue()" for items in AbortTransaction() to fix race + * prepare ABI for feature/socketpair + * Bump ABI to 4.15 + + [ David Kalnischkies ] + * reenable support for -s (and co) in apt-get source (Closes: 742578) + * run acquire transactions only once + * aborted reverify restores file owner and permission + * test if TMPDIR is accessible before using (Closes: 765951) + * chown finished partial files earlier + * promote filesize to a hashstring + + -- Michael Vogt Thu, 06 Nov 2014 10:01:21 +0100 + +apt (1.1~exp7) experimental; urgency=medium + + [ David Kalnischkies ] + * don't cleanup cdrom files in apt-get update (Closes: 765458) + * ignore Acquire::GzipIndexes for cdrom sources + + -- David Kalnischkies Wed, 15 Oct 2014 20:12:15 +0200 + +apt (1.1~exp6) experimental; urgency=medium + + [ josch ] + * implement the updated build profile spec + + [ Michael Vogt ] + * methods/rsh.cc: replace strcat with std::string (Closes: #76442) + * Add new configallowinsecurerepositories to the test framework + + [ Guillem Jover ] + * Update Status field values handling + + [ David Kalnischkies ] + * don't drop privileges if _apt has not enough rights + * check for available space, excluding root reserved blocks + + -- Michael Vogt Wed, 15 Oct 2014 07:47:36 +0200 + +apt (1.1~exp5) experimental; urgency=medium + + [ Michael Vogt ] + * Only rename StatError files in AbortTransaction() + * Document Acquire{MaxReleaseFileSize,AllowInsecureRepositories, + AllowDowngradeToInsecureRepositories} and + --no-allow-insecure-repositories + * Fix backward compatiblity of the new pkgAcquireMethod::DropPrivsOrDie() + * Change default of Acquire::AllowInsecureRepositories to "true" + so that this change is less disruptive, this will be switched + to "false" again after jessie + + [ David Kalnischkies ] + * remove useless pdiff filename output (Closes: 764737) + * make --allow-insecure-repositories message an error + * display a warning for unsigned repos + * trusted=yes sources are secure, we just don't know why + + -- Michael Vogt Mon, 13 Oct 2014 16:15:22 +0200 + +apt (1.1~exp4) experimental; urgency=medium + + [ Michael Vogt ] + * Merge sid version 1.0.9.2 + * feature/acq-trans: + - Make apt-get update more transactional by keeping all data from + a sources.list line in partial/ until all data is good and only + then move it into lists/ in one step + - add new -o Debug::Acquire::Transaction=1 debug option + * feature/expected-size: + Do not download more data in the mehotds than expected if we know + the size. For the InRelease/Release/Release.gpg add new + Acquire::MaxReleaseFileSize that defaults to 10Mb for now + * Verify the the hashes of the downloaded compressed files early + * Only load unauthenticated data into our parsers when the user + explicitly asked for it via --allow-insecure-repositories + (Acquire::AllowInsecureRepositories) + * Print warning when trying to use unauthenticated repositories + * Use /var/empty as the homedir for _apt + * Revert making pkgAcquire::Item::DescURI() "const" to not break + API + * Do not allow going from a authenticated to unauthenticated repository + * Add missing "adduser" dependency (for the new _apt user) + Thanks to Russ Allbery (Closes: #763004) + * Test if TMPDIR is a directory in apt-key and if not unset it + * add early verification for the .diff/Index download + * Bump library version to libapt-pkg4.14 + * Rework pkgAcqMeta{Index,Sig,ClearSig}::{Done,Failed]() for readability + * Ignore EINVAL from prctl(PR_SET_NO_NEW_PRIVS) (closes: 764066) + + [ David Kalnischkies ] + * deprecate Pkg->Name in favor of Grp->Name + * drop stored StringItems in favor of in-memory mappings + * de-duplicate version strings in the cache + * fix progress output for (dist-)upgrade calculation + * move PCI::From* methods into CacheSetHelper class (Closes: 686221) + * add a (hidden) --quiet option for apt-key + * only create new trusted.gpg if directory is writeable + * support (multiple) arguments properly in apt-key + * set a primary-keyring only if we have access to it + * merge fragment keyrings in apt-key to avoid hitting gpg limits + (Closes: 733028) + * use apt-key adv (+ gnupg) instead of gpgv for verify + * support gnupg2 as drop-in replacement for gnupg + * allow to specify fingerprints in 'apt-key del' + * use only one --keyring in gpg interactions + * add and use 'apt-key verify' which prefers gpgv over gpg + * remove empty keyrings in trusted.gpg.d on upgrade + * store source name and version in binary cache + * allow fetcher setup without directory creation (Closes: 762898) + * cleanup partial directory of lists in apt-get clean (Closes: #762889) + * allow options between command and -- on commandline + * update symbols file + * support parsing of all hashes for pdiff + * ensure world-readability for trusted.gpg in postinst (Closes: 647001) + * ensure partial dirs are 0700 and owned by _apt:root + * use _apt:root only for partial directories + * display errortext for all Err + * set PR_SET_NO_NEW_PRIVS also if run as non-root + + [ James McCoy ] + * ensure apt-key del handles 16-byte key ids (Closes: 754436) + + [ Kenshi Muto ] + * Japanese program translation update (Closes: 763033) + + [ Trần Ngọc Quân ] + * Set STRIP_FROM_PATH for doxygen + + [ Mert Dirik ] + * Turkish program translation update (Closes: 763379) + + [ Guillem Jover ] + * apt-get: Create the temporary downloaded changelog inside tmpdir + + [ Miroslav Kure ] + * [l10n] Updated Czech translation of apt (Closes: #764055) + + -- Michael Vogt Wed, 08 Oct 2014 09:37:35 +0200 + +apt (1.1~exp3) experimental; urgency=medium + + [ Michael Vogt ] + * merged changes from debian/sid up to 1.0.9.1 + * Make /var/lib/apt/lists and /var/cache/apt/archives owned + by the new _apt user + * Drop Privileges in the following acquire methods: + copy, http, https, ftp, gpgv, gzip/bzip2/lzma/xz + * DropPrivs: Improvements based on feedback from error@debian.org + + [ Julian Andres Klode ] + * DropPriv: Really call seteuid and not setuid, and add more checks + * Use _apt as our unprivileged user name + * DropPrivs: Also check for saved set-user-ID and set-group-ID + * methods: Fail if we cannot drop privileges + * DropPrivs: Also check for saved set-user-ID and set-group-ID + + -- Michael Vogt Wed, 24 Sep 2014 22:30:09 +0200 + +apt (1.1~exp2) experimental; urgency=medium + + [ Guillem Jover ] + * Add new Base256ToNum long long overload function + * Fix ar and tar code to be LFS-safe (Closes: #742882) + + [ Michael Vogt ] + * increase libapt-inst to version 1.6 + * Only allow "apt-get build-dep path" when path starts with ./ or / + * Allow passing a full path to apt-get install /foo/bar.deb (CLoses: #752327) + * merge changes from the 1.0.6 upload + + -- Michael Vogt Thu, 10 Jul 2014 13:18:08 +0200 + +apt (1.1~exp1) experimental; urgency=low + + [ David Kalnischkies ] + * [API Break] change "std::string pkgAcquire::Item::DescURI()" to + "std::string pkgAcquire::Item::DescURI() const" + * [ABI-Break] increase hashtable size for packages/groups by factor 5 + * [ABI-Break] cleanup datatypes mix used in binary cache + * [internal API-Break] remove the Section member from package struct + * use 'best' hash for source authentication (LP: 1098738) + * use HashStringList in the acquire system + * deal with hashes in ftparchive more dynamic as well + * reenable pipelining via hashsum reordering support + * parse and retrieve multiple Descriptions in one record + * improve pkgTagSection scanning and parsing + * invalid cache if architecture set doesn't match (Closes: 745036) + + [ Michael Vogt ] + * add support for "apt-get build-dep foo.dsc" + * add support for "apt-get build-dep unpacked-source-dir" + * add support for "apt-get install foo_1.0_all.deb" + * make "apt-get update" progress much more accurate by loading the + sizes of the targets into the fetcher early + * Implement simple by-hash for apt update to improve reliability of + the update. Apt will try to fetch the Packages file via + /by-hash/$hash_type/$hash_value if the repo supports that. + - add APT::Acquire::$(host)::By-Hash=1 knob + - add Acquire-By-Hash=1 to Release file + * add Debug::Acquire::Progress debug option + * [ABI-Break] lp:~mvo/apt/source-hashes: + - use sha{512,256,1} for deb-src when available LP: #1098738 + * [ABI-Break] stop exporting the accidently exported parsenetrc() symbol + * [ABI-Break] remove the PACKAGE_MATCHER_ABI_COMPAT defines + * [ABI BREAK] apt-pkg/pkgcache.h: + - adjust pkgCache::State::VerPriority enum, to match reality + * test/integration/test-debsrc-hashes: + - add integration test, thanks to Daniel Hartwig + * [ABI-Break] remove the PACKAGE_MATCHER_ABI_COMPAT defines + * [ABI-Break] Pass struct IndexTarget/indexRecords to + pkgAcqIndex{,Merge}Diffs + * [internal API-Break] rename pkgCache::Package::NextPackage to + pkgCache::Package::Next + * Calculate Percent as part of pkgAcquireStatus to provide a weighted + percent for both items and bytes + * apt-pkg/contrib/macros.h: bump library version to 4.13 + * apt-private/acqprogress.cc: do not show file size on IMSHit, it wasn't + fetched + * Fix warnings from clang -Wall/clang -fsanitize=address + * add DropPrivs() and drop privileges to nobody when running the + the buildin apt and dump solvers + * lp:~mvo/apt/webserver-simulate-broken-with-fix346386: + - fix invalid InRelease file download checking and add regression + test to server broken files to the buildin test webserver + - add regression test for LP: #34638 + + -- Michael Vogt Thu, 19 Jun 2014 12:01:48 +0200 + + apt (1.0.9.9) unstable; urgency=medium + + [ David Kalnischkies ] + * parse specific-arch dependencies correctly on single-arch systems + (Closes: 777760) + * remove "first package seen is native package" assumption. + Thanks to Axel Beckert for testing (Closes: 782777) + + -- David Kalnischkies Tue, 28 Apr 2015 16:11:27 +0200 + apt (1.0.9.8) unstable; urgency=medium [ David Kalnischkies ] diff --cc methods/https.cc index fa143439a,c97367323..d2ddf6fcf --- a/methods/https.cc +++ b/methods/https.cc @@@ -61,52 -53,25 +61,52 @@@ HttpsMethod::parse_header(void *buffer if (line.empty() == true) { - if (me->Server->Result != 416 && me->Server->StartPos != 0) + if (me->https->Server->Result != 416 && me->https->Server->StartPos != 0) ; - else if (me->Server->Result == 416 && me->Server->TotalFileSize == me->File->FileSize()) + else if (me->https->Server->Result == 416) { - me->Server->Result = 200; - me->Server->StartPos = me->Server->TotalFileSize; - // the actual size is not important for https as curl will deal with it - // by itself and e.g. doesn't bother us with transport-encoding… - me->Server->JunkSize = std::numeric_limits::max(); + bool partialHit = false; + if (me->Itm->ExpectedHashes.usable() == true) + { + Hashes resultHashes(me->Itm->ExpectedHashes); + FileFd file(me->Itm->DestFile, FileFd::ReadOnly); - me->https->Server->Size = file.FileSize(); ++ me->https->Server->TotalFileSize = file.FileSize(); + me->https->Server->Date = file.ModificationTime(); + resultHashes.AddFD(file); + HashStringList const hashList = resultHashes.GetHashStringList(); + partialHit = (me->Itm->ExpectedHashes == hashList); + } - else if (me->https->Server->Result == 416 && me->https->Server->Size == me->https->File->FileSize()) ++ else if (me->https->Server->Result == 416 && me->https->Server->TotalFileSize == me->https->File->FileSize()) + partialHit = true; + + if (partialHit == true) + { + me->https->Server->Result = 200; - me->https->Server->StartPos = me->https->Server->Size; ++ me->https->Server->StartPos = me->https->Server->TotalFileSize; + // the actual size is not important for https as curl will deal with it + // by itself and e.g. doesn't bother us with transport-encoding… + me->https->Server->JunkSize = std::numeric_limits::max(); + } + else + me->https->Server->StartPos = 0; } else - me->Server->StartPos = 0; + me->https->Server->StartPos = 0; - me->File->Truncate(me->Server->StartPos); - me->File->Seek(me->Server->StartPos); + me->Res->LastModified = me->https->Server->Date; - me->Res->Size = me->https->Server->Size; ++ me->Res->Size = me->https->Server->TotalFileSize; + me->Res->ResumePoint = me->https->Server->StartPos; - me->Res.Size = me->Server->TotalFileSize; + // we expect valid data, so tell our caller we get the file now + if (me->https->Server->Result >= 200 && me->https->Server->Result < 300) + { + if (me->https->Server->JunkSize == 0 && me->Res->Size != 0 && me->Res->Size > me->Res->ResumePoint) + me->https->URIStart(*me->Res); + if (me->https->Server->AddPartialFileToHashes(*(me->https->File)) == false) + return 0; + } } - else if (me->Server->HeaderLine(line) == false) + else if (me->https->Server->HeaderLine(line) == false) return 0; return size*nmemb; diff --cc methods/server.cc index bd01c3e98,6c05700a5..f61a6fedb --- a/methods/server.cc +++ b/methods/server.cc @@@ -314,20 -313,7 +324,20 @@@ ServerMethod::DealWithHeaders(FetchResu struct stat SBuf; if (stat(Queue->DestFile.c_str(),&SBuf) >= 0 && SBuf.st_size > 0) { - if ((unsigned long long)SBuf.st_size == Server->TotalFileSize) + bool partialHit = false; + if (Queue->ExpectedHashes.usable() == true) + { + Hashes resultHashes(Queue->ExpectedHashes); + FileFd file(Queue->DestFile, FileFd::ReadOnly); - Server->Size = file.FileSize(); ++ Server->TotalFileSize = file.FileSize(); + Server->Date = file.ModificationTime(); + resultHashes.AddFD(file); + HashStringList const hashList = resultHashes.GetHashStringList(); + partialHit = (Queue->ExpectedHashes == hashList); + } - else if ((unsigned long long)SBuf.st_size == Server->Size) ++ else if ((unsigned long long)SBuf.st_size == Server->TotalFileSize) + partialHit = true; + if (partialHit == true) { // the file is completely downloaded, but was not moved if (Server->HaveContent == true) diff --cc methods/server.h index 1b1f754a3,8c14282b6..8d7d33ee6 --- a/methods/server.h +++ b/methods/server.h @@@ -72,12 -77,11 +79,12 @@@ struct ServerStat }; /** \brief Get the headers before the data */ RunHeadersResult RunHeaders(FileFd * const File, const std::string &Uri); + bool AddPartialFileToHashes(FileFd &File); bool Comp(URI Other) const {return Other.Host == ServerName.Host && Other.Port == ServerName.Port;}; - virtual void Reset() {Major = 0; Minor = 0; Result = 0; Code[0] = '\0'; Size = 0; JunkSize = 0; + virtual void Reset() {Major = 0; Minor = 0; Result = 0; Code[0] = '\0'; TotalFileSize = 0; JunkSize = 0; StartPos = 0; Encoding = Closes; time(&Date); HaveContent = false; - State = Header; Persistent = false; Pipeline = true;}; + State = Header; Persistent = false; Pipeline = true; MaximumSize = 0;}; virtual bool WriteResponse(std::string const &Data) = 0; /** \brief Transfer the data from the socket */ diff --cc test/integration/test-apt-download-progress index 07c5e09c5,7092e163e..65c438e8f --- a/test/integration/test-apt-download-progress +++ b/test/integration/test-apt-download-progress @@@ -34,9 -34,10 +34,8 @@@ testsuccess --nomsg apthelper download- assertprogress apt-progress.log msgtest 'download progress works via' 'https' -printf '\n' exec 3> apt-progress.log -testsuccess apthelper download-file "https://localhost:4433/$TESTFILE" https-$TESTFILE -o APT::Status-Fd=3 -o Acquire::https::Dl-Limit=600 -assertprogress apt-progress.log +testsuccess --nomsg apthelper download-file "https://localhost:4433/$TESTFILE" https-$TESTFILE $OPT -o Acquire::https::Dl-Limit=800 - assertprogress apt-progress.log # cleanup rm -f apt-progress*.log diff --cc test/integration/test-bug-lp1445239-download-loop index 000000000,342e3c61e..4e4de67bd mode 000000,100755..100755 --- a/test/integration/test-bug-lp1445239-download-loop +++ b/test/integration/test-bug-lp1445239-download-loop @@@ -1,0 -1,29 +1,29 @@@ + #!/bin/sh + # + # this is a regression test for LP: #1445239 where a partial download can + # trigger an endless hang of the download method + # + + set -e + + TESTDIR=$(readlink -f $(dirname $0)) + . $TESTDIR/framework + setupenvironment + configarchitecture 'amd64' + + changetowebserver + webserverconfig 'aptwebserver::support::range' 'true' + + TESTFILE='aptarchive/testfile' + dd if=/dev/zero of=$TESTFILE bs=100k count=1 2>/dev/null + + DOWNLOADLOG='rootdir/tmp/testdownloadfile.log' + -TARGET=testfile-downloaded ++TARGET=./downloaded/testfile-downloaded + dd if=/dev/zero of=$TARGET bs=99k count=1 2>/dev/null + if ! downloadfile http://localhost:8080/testfile "$TARGET" > "$DOWNLOADLOG"; then + cat >&2 "$DOWNLOADLOG" + msgfail + else + msgpass + fi