Database disk image is malformed

[Jon Polish]

Originally posted by kinook
After a word has been unchecked from all items, it will no longer appear in the list (after the Item Keywords dialog is dismissed and then displayed again).

Wow, I didn't know that. I will test this out as soon as my database finished repairing and compacting (probably in the next 7 hours - no joke!). I had to kill a mbx import operation last Friday. When I attempted to import this morning, UR reported that the database needed repair. So I repaired and compacted and am growing old waiting for it to finish.

Seriously, it is frustrating that this takes so long (importing too) which means I cannot have access to my data.

Jon

[Jon Polish]

In http://www.kinook.com/Forum/showthread.php?s=&threadid=2563 I mentioned that I had to repair one of my databases. I performed a repair and compressed it. Although it took forever, it completed the operations without incident. Apparently my database did not require compressing since the size did not change. That was a step I could have avoided and would have sped up the process. Live and learn.

Anyway, I now am trying to import additional email via mbx import. The process chugs along and then I get this error:
Error importing files: Database disk image is malformed

The import fails of course. I am trying to add approximately 1200 emails with attachments. All previous imports to this database have been successful.

Here is the UR information:
Ultra Recall Professional 3.2.5
Registered to: Jon Polish (1-user license)
Windows version: 5.1.2600.2.0
Install path: C:\Program Files\Ultra Recall
EncryptPDF.dll version 3.0.0.2
mimepp.dll version 3.0.4
pdf2txt.dll version 3.1.0.4
PolarSpellChecker.dll version 4.0.5.4
riched20.dll version 5.50.99.2010
SftPrintPreview_IX86_U_10.dll version 1.05
SftTree_IX86_U_50.dll version 5.06
UltraRecall.exe version 3.2.5.2
unins000.exe version 51.48.0.0
Database filename: C:\Documents and Settings\Jon Polish\My Documents\Ultra Recall\Email archives.urd
Database version: 3.0.13

Please see the attachments for additional information.

Thanks for any assistance.

Jon

[kinook]

http://www.kinook.com/Forum/showthread.php?threadid=2673

[Jon Polish]

Thanks. Not what I wanted to hear, because the time involved in copying to a new database, repairing with your tool, or re-importing to a new database will be prohibitive.

For your information, this probably happened because I had to kill UR during an very long import operation. I needed information from another database and could not terminate the import normally. Anyway, I looked for the temp files (journal), but they were nowhere to be found. I guess this is why I got the error.

Jon

[Jon Polish]

I have decided to copy the data from the damaged database to a new one. The new database displays the Copying Items dialogue, but then the old database displays the Server Busy dialogue (which I thought was a thing of the past). Once this occurs, nothing happens. Any suggestions?

Jon

[kinook]

With that large of a database, method #3 will probably be a lot faster than #2.

[Jon Polish]

Originally posted by kinook
With that large of a database, method #3 will probably be a lot faster than #2.

OK, but what about this "Server Busy" message?

[kinook]

I'm not sure. I couldn't reproduce that behavior with a long-running inter-db copy operation (although the copy operation is likely continuing, and it's probably not a good idea to try to do anything in the database during the copy operation anyway). What triggered it?

[Jon Polish]

There were two databases open. The damaged one, and the new default database. They are shown in the screenshot in my previous post. All I did was drag a parent item with a number of children. In my first attempt, I tried to copy all of the received email (there are many tens of thousands). Subsequent tries with smaller and smaller amounts of items yielded the same results. The copying dialogue appeared then shortly after the server busy box appeared as shown. While I could dismiss the server busy box by clicking switch to or retry, it re-appeared in very short order. The only way to get rid of it was to cancel the copy dialogue.

By the way, the repair tool is currently doing its thing. You were correct, it is a much faster way to go. I'll let you know if it repairs the corruption.

Jon

[ashwken]

Originally posted by Jon Polish
The copying dialogue appeared then shortly after the server busy box appeared as shown. While I could dismiss the server busy box by clicking switch to or retry, it re-appeared in very short order. The only way to get rid of it was to cancel the copy dialogue.
I experienced the same thing over the weekend.

Was copying a number of nodes (stored web pages) from one database to another - some of the nodes had child nodes (of stored web pages). When the dialog appeared I click on Switch To and the dialog eventually went away, then would reappear, and I would click on Switch To...

The appearance of the dialog seemed to coincide with the number of nodes (not including child nodes) I had selected to copy. What concerned me was not knowing if the dialog really needed a response, to allow the copying to continue, or if the copying was proceeding regardless of the presence of the dialog.

Anyway, just my two cents.

[Jon Polish]

Originally posted by Jon Polish
By the way, the repair tool is currently doing its thing. You were correct, it is a much faster way to go. I'll let you know if it repairs the corruption.

This seems to have done the trick. Initially, the repair proceeded relatively quickly through step two. I was very pleased. Then in the last stages of step 2 (the remaining 400MB of the file), it just crawled. When I left the office for the night, it had just under 400MB left. Upon return the next morning (6:40am) about 200MB remained. The process finally finished at 12:15pm. That is a very long stetch for just under 200MB. Maybe that was the part of the file that was corrupted so it took longer.

Jon

[Jon Polish]

Kinook:

Can you please describe what this repair utility does? The reason I ask is because I have noticed greatly improved performance with this database. The compress and repair built into UR does nothing that I can tell to improve performance. I notice that the repaired file is a bit smaller. Is there any way of knowing what changes were made?

Thank you.

Jon

[kinook]

The regular compact+repair routines assume that the underlying database file is intact, but may be fragmented and/or contain orphaned records, incorrect computed values, or other SQL errors within the stored tables.

A "malformed database" error indicates that the SQLite database engine is unable to read part of the database file, which means that the actual file has been corrupted on disk. Normally when this error is reported, a regular compact/repair will also fail, as it performs a scan of the entire .urd file, which will encounter any bad block(s).

The external repair tool works similar to a compact operation (read the entire database and completely recreate it without unused blocks), but it is able to skip bad blocks, which allows the operation to complete even when the actual .urd file is corrupted. I'm not really sure how performance could be significantly improved using this method vs. regular compact.

[Jon Polish]

Originally posted by kinook
I'm not really sure how performance could be significantly improved using this method vs. regular compact.

I certainly can't know why if you are without an explanation. However, I can report that the import process which I could not perform because of the fragmentation now proceeds quicker. I can also say that the import process is quicker than ever - even when I first created the database. Trust me, this is not my wishful thinking. I have complained about performance in the past. I am not experiencing what I would call speedy performance, but it is a very noticable improvement.

Jon

[janrif]

Originally posted by kinook
[snip] The external repair tool works similar to a compact operation (read the entire database and completely recreate it without unused blocks), but it is able to skip bad blocks, which allows the operation to complete even when the actual .urd file is corrupted. I'm not really sure how performance could be significantly improved using this method vs. regular compact.
What / where is this external repair tool that you reference?

[Jon Polish]

http://www.kinook.com/Forum/showthread.php?threadid=2673

[janrif]

Originally posted by Jon Polish
http://www.kinook.com/Forum/showthread.php?threadid=2673
Thank you. Jon. I'm thinking that the next time I get an error I might run this instead of the internal tool. I've reported a couple of non-fatal errors which might be helped -- just guessing but then I wonder if I'm not looking for trouble for a relatively minor issue. ??

[Jon Polish]

Originally posted by janrif
just guessing but then I wonder if I'm not looking for trouble for a relatively minor issue. ??

I don't think it is a destructive process at all Jan. I wish Kinook provided me with the answer to my question about a way to know what specific changes were made, but I don't see any loss of data. If you are concerned, you can always make a copy of your database and perform the repair on the copy.

Jon