If it's taking a lot longer than that, certainly 24 hours, then it must be downloading from a very slow node, or your connection is much slower than around 15KB per sec (120kbps), or something else is wrong. It would be nice to know what appears to be the bottleneck when that happens.
Every 10 minutes or so when the latest block is sent, it should have the chance to change to a faster node. When the latest block is broadcast, it requests the next 500 blocks from other nodes, and continues the download from the one that sends it fastest. At least, that's how it should work.
Someone should experiment with different Berkeley DB settings and see if there's something that makes the download substantially faster. If something substantial is discovered, then we can work out the particulars.