Could something like this be done or did I watch too many movies?

[Thee]

Hey

Got an idea yesterday, that maybe some programmer could answer me best. I need thinking out of the box for a second.
Maybe you will laugh, but what the heck, if this is not possible, at least I'll be a little smarter after the discussion or maybe start a good concept for a SF movie

Anyway, my idea is about "downloading" (copying a file) and possibility to speed things up only using software, and not upgrading hardware or your internet connection.

So, how this would work...

As I understand, when you download a file, the download manager reads the file byte by byte and copies it to your HDD. Is that correct? If it is, read on...

What if you didn't need to download the actual file? What if the download manager can make an exact local copy of the file you want to download only by getting some information from the Server about the file.

How this would work:
The Client sends the download request for a file to the Server. Then the Server analyses that file and says to the Client: Im not going to send you such a large file psychically, it will take too much time and bandwidth, instead Im gonna send you signals that will perfectly describe how this file looks like, and then you can use this information that I give you and make a local copy of the same file that exists here.

So it's like this: don't send me a psychical file, send me a description of that file, and I will make it psychical here locally.

I have no idea really how that would work, maybe the Server would say "ABC" and the client would know he has to write some information that code "ABC" represents.

So instead of downloading 1 MB of some data, the server would send a code "ABC" (for example) and the client would know what that means, and make an exact copy of that 1 MB of data. So if sending the information "ABC" is quicker then downloading 1 MB of actual data, then you would "download" that 1 MB of data faster with the same internet connection, thus boosting the speed needed to copy something over the internet.

Now this probably is not possible, but I want to know why?

And if there is some kind of workaround to achieve something similar to this. I mean, do we really need to download something? Cant we just clone it locally based on information sent? Or would that get to a point of diminishing returns?

So fiction or plausible?

[MG&TL]

Perfectly plausible, this is what happens (If I understand you correctly) with compression, i.e. downloading a zip file. Problem is, compression isn't that good, so "ABC" wouldn't work too well.

[dannyboy79]

i don't think this would work for say a movie, if you wanted to download a movie, theres no amount of data the server could provide for the client to "make" that movie locally.

cool idea however

[Paqman]

Yep, you've just described compression.

Say you have a file containing the following data:

AAABB BBCCC CCDEE (15 "bits")

A compression algorithm might shorten that down to:

3A 4B 5C D 2E (9 bits)

The compressed file doesn't contain the original data, it contains "signals that will perfectly describe how this file looks like" as you put it.

Compression is used to speed up connections. It's used on the server side somewhat, and the old Opera Mini browser for phones used to compress all traffic to try and make browsing on a nasty GPRS connection tolerable.

[Thee]

Ok, so I invented ZIP, great '89 here I come.

So basically what you guys are saying is, we are already achieving this by using compression, there is no other magical way to do this stuff better then what we are already doing?

[koenn]

So basically what you guys are saying is, we are already achieving this by using compression, there is no other magical way to do this stuff better then what we are already doing?

Well, 2 things
1-
"compression" is a broad generic term. There are many compression algorithms. It's not impssoble to come up with a new one

2- hash/fingerprint functions, maybe ?
Theoretically, each file produces a unique hash.
So if the server sends you a hash of the file you requested, all you have to do is recreate a file that, given the function, creates the same hash. You then know that the file you created is identical to the one on the server.

Unfortunately, creating the correct file may be difficult.

[Thee]

Well, 2 things
1-
"compression" is a broad generic term. There are many compression algorithms. It's not impssoble to come up with a new one

Yeah I know that, but its not possible in a way that I described, or it is possible but not practical enough.

2- hash/fingerprint functions, maybe ?
Theoretically, each file produces a unique hash.
So if the server sends you a hash of the file you requested, all you have to do is recreate a file that, given the function, creates the same hash. You then know that the file you created is identical to the one on the server.

Unfortunately, creating the correct file may be difficult.

Not really sure how that works, but it sounds interesting.
Is there any "proof of concept" on a small scale for those functions used in the matter I described?

[MG&TL]

Not really sure how that works, but it sounds interesting.
Is there any "proof of concept" on a small scale for those functions used in the matter I described?

"One-way" compression is often a good way to check file integrity, i.e. MD5sums.

You can generate a unique hash from a file like this:

md5sum <file>

...but try recreating that file and it's (hypothetically at least) a one-way process without a lot of time and computing power.

[squenson]

You can have a look at fractal compression. The benefits are very large compression factors, unfortunately, the time to compress a file is extremely long.

[koenn]

... without a lot of time and computing power.

yeah, in a worst case scenario you'd have to brute-force it, ie start from an empty file and create all possible files until one checks out.


otoh, iirc, rsync uses a sort of "reuse parts of existing files on the destination system to reduce data transfer" approach. It's mostly used to "sync" two copies of supposedly the same file, but I think such algorithm could also be applied between unrelated files. Like "rsync" (remote) picture A against (local) picture B, to convert picture B into (local) picture A.
The problem then becomes : how to find a suitable file.

rsync --fuzzy seems to be going in that direction



And in the end it's always a trade off between bandwidth and processing, and their relative costs.