Lars Pensjö
September 4th, 2011, 10:10 PM
Background:
Application with several millions of objects.
Objects vary in size from 300 bytes to 32000 bytes.
Every object is identified by a unique 24 byte key.
There is only one application, but many threads that access the data.
Approximately 100 read operations for every write.
Any suggestions on how to best manage this data? Currently, it is all saved as individual files in one single directory, with the file name as a representation of the key. But I don't know how well ext4 scales with the number of files.
An obvious answer might be MySQL. I would guess it is slower than ext4 for a "small" umber of objects.
I suppose ext4 has 4kB blocks, which means there is a waste. If this waste is only in space, not in time, then the trade-off is acceptable.
Application with several millions of objects.
Objects vary in size from 300 bytes to 32000 bytes.
Every object is identified by a unique 24 byte key.
There is only one application, but many threads that access the data.
Approximately 100 read operations for every write.
Any suggestions on how to best manage this data? Currently, it is all saved as individual files in one single directory, with the file name as a representation of the key. But I don't know how well ext4 scales with the number of files.
An obvious answer might be MySQL. I would guess it is slower than ext4 for a "small" umber of objects.
I suppose ext4 has 4kB blocks, which means there is a waste. If this waste is only in space, not in time, then the trade-off is acceptable.