[IDEA] Delta (patch based) updates

[zolookas]

Summary:
Ability to download only changed bits of files and use much less bandwidth.
Scope and Use Cases:
Ann has slow internet connection. She sees that there are 150MB of updates and decides not to update at all leaving her with vulnerable and buggy system.
Implementation Plan:
Adopt it from Debian?

Previously discussed here, but still not implemented: http://ubuntuforums.org/showthread.php?t=409916

[ravirdv]

+1 coz it would save much time & bandwitdh

[artir]

We really need this!
+1

[ccw]

Implementation Plan:
Adopt it from Debian?


https://blueprints.launchpad.net/ubuntu/+spec/apt-sync

The spec referenced in the other thread is for apt-sync which in debian grabs incremental diffs of package LISTS in apt-get update, not the diffs of packages themselves.

[FredB]

I saw this for OpenSuSE 10.3. It could be a great idea for saving time on updates.

[Mathiasdm]

There've been requests for delta updates for quite a while now, it's about time they got implemented.
Especially while running development versions, it often happens that one has to download 100 MB of updates in a day.

@ccw: I thought apt-sync did work for packages AND lists, but pdiffs (implemented in apt) worked only for links (source: the link you provided ;) ).

[ccw]

@ccw: I thought apt-sync did work for packages AND lists, but pdiffs (implemented in apt) worked only for links (source: the link you provided ;) ).

Well, the only thing I've seen in Debian is pdiff. Maybe I'm missing something.

[reacocard]

+1, smaller updates are always good.

[Zdravko]

+1. I'd love this one!

[Next.Step]

+1
pleeeeease!

[Mathiasdm]

I'm going to bump this back to the top. It's a brilliant idea, and I think a lot of people would love it.
It's just something most people agree with, and that's probably why there's not much discussion on it (why discuss if people agree :p ).

[ravirdv]

ya obviously everyone wants this to be implemented:guitar:

[CrazyGuy123]

Doesn't it require a change in the way packages are packaged because you can't do a decent diff on compressed files, so you've gotta package all your .debs uncompressed, which in turn means they take more disk space.

It also means to have something to diff agains you need to cache the original .debs for every package installed, which wastes a lot of disk space.

[robinl]

+1 , you can just make it as a package of diff files instead of the actual files, and a programme to apply that diff to the installed files.

[drf_av]

+1.
Fedora has almost implemented this, so did with PulseAudio, and probably will be implementing PackageKit. Looks like we're a step behind.

[nocturn]

-1

Yes, SuSE does this and it is much slower then before on systems with a good network connection.

I think we need to look first at the size of bandwith saving and only continue if it is significant, which I do not think.

[coolen]

If the package is just a collection of diffs, what happens if you're installing, not upgrading?
As previously mentioned, this would also mean you'd need to cache all packages to make it worthwhile.

I'm gonna vote this down. It sounds great in theory, but in practice, it represents a lot of work, and probably a fair amount of forkage.

[drf_av]

@coolen: you're right, actually. The best implementation is having a repo for installing, such as the one we have now, and a repo for delta updates, which, I think, it's the way PrestoRPM works now. I could be wrong about Presto, but I really think this is the only way to implement this.

Also nocturn is right: some benchmarks about a possible solution would be nice but... I wonder if something like this for apt already exists? Maybe in Debian?

[coolen]

A project called apt-sync was mentioned above (which I'm assuming is based on rsync).
This would be nice, don't get me wrong, but to include it by default, as part of the package manager, would require a lot of work on APT directly. This is a little beyond the Ubuntu team's reach.

[CrazyGuy123]

rsync is too CPU heavy, and therefore would make the process too slow. From theory, I guess it takes a lot of CPU to create the diffs, but not much CPU to apply them - so really we only want to create the diffs once on the server and then all that needs to happen on the client side is applying the diff.

The idea of diffing the installed files rather than the .debs is good, but it's hard to ensure the installed files are pristine, and havn't been tampered with by a third party program. If they have been tampered with it means you've got to download a complete package during installation, and if the download fails you've left the system in a broken state (because the package pre-install scripts have already been run, but the actual files haven't been installed)

[vijaym]

I like the idea. it would save me lots of bandwidth

[Slug71]

+1

[gnomeuser]

rPath' Conary has a very sexy implementation of this. It also does rollback and other very sexy things. Maybe it's time to realise that moving on from dpkg and apt is a real chance for innovation and to bring us much needed features. It also brings the concept of a distributed version control system into the package manager which is very helpful for things like appliances and specialized deployments (and who can deny Ubuntu is going that way already). All conary reciepes are basically little python scripts, most are autogenerated using conary tools making maintance a breeze compared to the current generation of packaging formats.

This would naturally be a multi cycle effort and it would require either convincing debian to consider working forwards this goal as well or rebasing to rPath.

I know that is a lot of work but even Debian has been considering a next generation packaging system for a while. Conary is a very good proposal for such a job and now with PackageKit we don't need to rewrite our frontends, just make them call PackageKit instead of the PMS API directly and you get to keep that work. If we wanted such a migration a firm roadmap would need to be formulated but it's entirely possible.

It will take time and we will need to reeducate existing packagers to use the new format. It's not without downsides but I think it's definitely worth considering.

Background reading:
http://en.wikipedia.org/wiki/Conary_(package_manager)
http://wiki.rpath.com/wiki/Conary

Video presentations (FOSDEM 2008):
http://video.fosdem.org/2008/maintracks/FOSDEM2008-conary.ogg

[Changturkey]

rPath' Conary has a very sexy implementation of this. It also does rollback and other very sexy things. Maybe it's time to realise that moving on from dpkg and apt is a real chance for innovation and to bring us much needed features. It also brings the concept of a distributed version control system into the package manager which is very helpful for things like appliances and specialized deployments (and who can deny Ubuntu is going that way already). All conary reciepes are basically little python scripts, most are autogenerated using conary tools making maintance a breeze compared to the current generation of packaging formats.

This would naturally be a multi cycle effort and it would require either convincing debian to consider working forwards this goal as well or rebasing to rPath.

I know that is a lot of work but even Debian has been considering a next generation packaging system for a while. Conary is a very good proposal for such a job and now with PackageKit we don't need to rewrite our frontends, just make them call PackageKit instead of the PMS API directly and you get to keep that work. If we wanted such a migration a firm roadmap would need to be formulated but it's entirely possible.

It will take time and we will need to reeducate existing packagers to use the new format. It's not without downsides but I think it's definitely worth considering.

Background reading:
http://en.wikipedia.org/wiki/Conary_(package_manager)
http://wiki.rpath.com/wiki/Conary

Video presentations (FOSDEM 2008):
http://video.fosdem.org/2008/maintracks/FOSDEM2008-conary.ogg
Debian is considering a new package system?

[edajai]

debian based system really needs a delta upgrade system. This is one thing where rpm is really leading. Opensuse has support for it since 10.3 and Fedora 11 is having something similar called Presto (plugins for it is available for fedora 9 and 10 also). I really hope something concrete about this is discussed in the coming UDS-Jaunty.

[shae]

rPath' Conary has a very sexy implementation of this. It also does rollback and other very sexy things. Maybe it's time to realise that moving on from dpkg and apt is a real chance for innovation and to bring us much needed features. It also brings the concept of a distributed version control system into the package manager which is very helpful for things like appliances and specialized deployments (and who can deny Ubuntu is going that way already). All conary reciepes are basically little python scripts, most are autogenerated using conary tools making maintance a breeze compared to the current generation of packaging formats.

This would naturally be a multi cycle effort and it would require either convincing debian to consider working forwards this goal as well or rebasing to rPath.

I know that is a lot of work but even Debian has been considering a next generation packaging system for a while. Conary is a very good proposal for such a job and now with PackageKit we don't need to rewrite our frontends, just make them call PackageKit instead of the PMS API directly and you get to keep that work. If we wanted such a migration a firm roadmap would need to be formulated but it's entirely possible.

It will take time and we will need to reeducate existing packagers to use the new format. It's not without downsides but I think it's definitely worth considering.

Background reading:
http://en.wikipedia.org/wiki/Conary_(package_manager)
http://wiki.rpath.com/wiki/Conary

Video presentations (FOSDEM 2008):
http://video.fosdem.org/2008/maintracks/FOSDEM2008-conary.ogg

No. Just simply no. Have you ever actually used conary? When I first heard about it, I thought it was an awesome idea. I tried it (with Foresight Linux) and found it intolerably slow. The generation of diffs on the user side requires so much time that it quickly outweighs any time improvements on the side of downloading packages. Perhaps someone with a metered connection who can only download a certain amount would find this acceptable, but anyone with a reasonable unlimited connection would find this ridiculous. There are much easier ways to trim bandwidth consumption such as breaking up packages more so they do not have to be constantly updated if one part is updated or patched (such as l-r-m) and better update quality assurance to prevent having to release multiple updates to the same package in the same week.