importing many emails

[kzerba]

(1) converted ~2500 thunderbird emails to .eml files (time: ~10 min)
(2) imported into UR (time ~4 hrs)
(3) deleted same emails from UR (time ~ 4 hrs)

(2) and (3) pretty required not doing anything else on this computer during this time (IBM T30, 512 mb ram, WinXp SP2)

Is there any way to make this kind of operation faster?

kzerba

[dasymington]

It would be good if it could be faster. I find even importing one or two .eml files can take up to 30 seconds or so. It must be because UR has to do a lot when it's importing - keywording, etc.

Presumably you'd only import such a large number initially and then you'd just be topping up with new messages, and normally you wouldn't be deleting so many at a time.

[kinook]

There is a fair amount of overhead to parse each message (and attachments) for keywords and other info, and supporting undo/redo also adds overhead. Some things that could speed up importing would be disabling the options under 'Tools | Options | Import (More) | File Import' and/or similar options on the Import Wizard (depending on how you are importing). Of course, you'll lose some functionality by doing that. Encrypting your database also adds some overhead.

But that averages out to about 10 seconds per message, which seems high, and 30 seconds is excessive to import just one or two. How large are the messages being imported? Are the databases located on network shares or local paths? What kind of hardware (CPU & HD) are your systems? I have an AMD 64 x2 Dual Core desktop w/ 2GB RAM and RAID0 HD, and I averaged about 3 messages per second to import 350 messages (ranging in size from 1K to 175K) from Outlook Express to a 100MB unencrypted database on a local drive (and 15 seconds to permanently delete).

[kzerba]

Some of the 2500 emails were large. Email is one of my biggest headaches and I was experimenting with possibly scaling up in use of UR to manage my emails more effectively and reduce the headache. I don't want to lose the functionality of the import options to enhance the speed of the import.

Import context: IBM T30 laptop, Pentium M 1.5GHz processor, 512 mb ram, 37 GB hd with 7 GB free, local paths, no encryption, urd file size 32 mb.

To make an import comparable to your 350 email example, I selected emails < 200 kb from the 2500 and then took a representive sample of 350 from those (size 1-192 kb). This import took 15.5 min or 1 email every 2-3 sec, which is closer to your result.

So, size does matter, but so do the obvious things such as processor, ram, hd speed, etc. This experiment has taught me some things that I will consider in deciding how many emails at one time, which sizes of email and when to import to UR.

One thing I noticed is that if I import as a folder with the emails in the folder, UR always leaves one email in the folder that was imported. If I import as files just selecting the emails from the folder all is fine.

kzerba

[dasymington]

Sorry for hijacking the thread; it's about importing many emails whereas I was talking about a few emails at a time. UR does import messages faster when they're in bulk, as you can see from the figures below.

My "30 seconds or so" was a bit subjective: a measured test shows that importing 3 .eml messages (4.30, 7.12 and 2.12 KB respectively) took 20 seconds. This is dragging the messages from Explorer and dropping them onto a folder in UR. The UR database is on a local drive. The PC is a 3.2GHz Pentium 4 with 1 GB RAM. The local partition on which the UR database resides is 97 GB with 60 GB free. The UR database is 655 MB and unencrypted.

Doing a bulk import in the same way for 60 messages (average 3 KB), took 1 min 30 secs. So that's about 6 seconds per message when only a few messages and 1.5 seconds per message when there's a larger number. No doubt the average time would drop if there were 100s or 1000s of messages.

Six seconds a message is perhaps not much, but it can seem a while the way I work, which is basically to store all important message in UR at various points throughout the day as I deal with them.