Books disappear after calibre crash

[Giuseppe Chillem]

Hello,
I have added some books to my library when the TAG reader crashed.
I have then read manually the image of the books whose name was "unknown" via "Modify Metadata Individually". After 8 books I have encountered the "problematic" pdf and calibre crashed.
After having realoaded calibre all the books I have edited before the problematic one disappeared. Where are they ?

As I encounter often this kind of problem I have a little request: could please make visible the TIME the book has been added to the library ? This would help to intercept and delete all the books added during the last run. Note, the time should be the same for all the books added in a single batch.

(Waiting for the crc32 + filesize duplicate detection...)

Giuseppe Chillemi

[Starson17]

Originally Posted by Giuseppe Chillem

After having realoaded calibre

Did you restart Calibre or actually have to reinstall it?

all the books I have edited before the problematic one disappeared. Where are they ?

Did you look in the Library?

As I encounter often this kind of problem I have a little request: could please make visible the TIME the book has been added to the library ? This would help to intercept and delete all the books added during the last run.

It appears you know this, but if not, the time is stored, but not shown. Sorting by date also sorts by time.

Note, the time should be the same for all the books added in a single batch.

I disagree. I prefer to have it use the actual time added, which is what it does.

(Waiting for the crc32 + filesize duplicate detection...)

I have considered working on metadata-based (author/title and isbn) duplicate detection, but not crc32 or filesize - neither of which is of interest to me. You already know you can do this with 3rd party tools.

[Giuseppe Chillem]

1) I have restart Calibre without reinstalling.

2) I have sorted the library by date and they were not there.

3) I know the time is there but I want to see it. It will solve once and for all the problem of multiple batch of books added the same day. In the actual scenario I can't recognize where one batch end and where another starts.

4) Can't understand why the precise adding time is better for you than the batch adding time (one and for all). The former doesn't give us any usefull information, the latter let us identify the latest batch.

5) File Hash + Duplicate tag is the complete solution for any problem. When a duplicate file hash is found you have 99.99999999% chance you have a duplicate. Using other tools complicate things too much as new files needs the duplicate library to be rebuild, even for one file added.

The final solution is:

File Hash duplicate check -> Ask for deletion.
Then:
Start the current Tag based duplicate check.

Berlieve me when I say your library will never have duplicates and believe me when I say this solves the Calibre Crash problem that generates duplicates. If the TAG library crashes and you have 100 "unknown - Unknown" books the tag mechanism won't work while the hash mechanism will find 100% of them, either checking against existing books or if you restart the batch, against new added books.

Regards
Giuseppe Chillemi

[Starson17]

Originally Posted by Giuseppe Chillem

1) I have restart Calibre without reinstalling.

Good. I've never needed to reinstall (and I can't recall any crashes either, not that I disbelieve you.)

2) I have sorted the library by date and they were not there.

I was suggesting you use your OS file browser to look directly in the folder set as the Calibre Library folder. It should have your books. I have no idea why they don't show up in Calibre.

3) I know the time is there but I want to see it.

I understood that.

It will solve once and for all the problem of multiple batch of books added the same day. In the actual scenario I can't recognize where one batch end and where another starts.

I have no trouble identifying the breaks. The gap in time is much larger. I view this info in an SQL Browser when I want it.

4) Can't understand why the precise adding time is better for you than the batch adding time (one and for all). The former doesn't give us any usefull information, the latter let us identify the latest batch.

I don't see how you can identify the batch if you don't see the time. I can identify both, but I have to use the SQL browser. I've occasionally needed to know how long it takes to add files. I can get that from the exact time during a batch add.

5) File Hash + Duplicate tag is the complete solution for any problem. When a duplicate file hash is found you have 99.99999999% chance you have a duplicate.

True, but not finding a duplicate doesn't mean you don't have any duplicates, and I'm almost always looking for duplicates, not 99.99999999% duplicates. Tiny changes in the file mean that I wouldn't get a match with file hash/CRC. It's just not a tool I need. If I did need it, I'd use a third party file dupe finder. I'm not saying you don't need/want it.

The final solution is:

File Hash duplicate check -> Ask for deletion.

I keep certain 100% duplicates I don't want deleted. Specifically, If I want 2 different copies of a particular format, I usually copy the record, with all formats, then change only the one format.

believe me when I say this solves the Calibre Crash problem that generates duplicates.

I've never had a duplicate due to a "Calibre Crash."

If the TAG library crashes and you have 100 "unknown - Unknown" books the tag mechanism won't work while the hash mechanism will find 100% of them, either checking against existing books or if you restart the batch, against new added books.

I don't fully understand this. If you are getting a crash using tags, you should report it. I've never seen one. I have lots of "Unknown" authors, but they are truly unknown and they aren't duplicates very often. IIRC from your earlier posts, you are seeing "Unknown" because you have metadata pulled from inside the file, and the internal metadata isn't there, or it's poor. I can understand in that case that you want hash-based duplicate detection, as your metadata is so poor, but it's just not something that I would ever use. My books almost always have some identifying author or title information in the filename to work with.

[speakingtohe]

@Giuseppe Chillemi
Can't you search for unknown?

File Hash duplicate check -> Ask for deletion.
Then:
Start the current Tag based duplicate check.

Berlieve me when I say your library will never have duplicates and believe me when I say this solves the Calibre Crash problem that generates duplicates. If the TAG library crashes and you have 100 "unknown - Unknown" books the tag mechanism won't work while the hash mechanism will find 100% of them, either checking against existing books or if you restart the batch, against new added books.

How can this work? If you modify a book (even by importing to calibre) by adding metadata, changing title etc. the hash will change. The hash would have to be calculated before importing which would possibly slow things down on an overall level.

Even a hash based on title only will be useless as many titles seem to be chosen by different authors. Title-author, you have to deal with things like spaces after initials or periods after initials.

For example J. D. Robb could be written in many ways
J. D. Robb
J.D. Robb
J D Robb
Norah Roberts as J. D. Robb.

Only hash code that would actually work is one generated on original file and this could not be done until file was added.


While pretty old school, hash codes still have their place, but I don't think it is a viable solution in this instance. The current duplicate detection is pretty good although not foolproof.

Of course you can prove me wrong by writing a foolproof hash code based calibre module to do this

Regards
Helen

[Giuseppe Chillem]

Sorry17 Starson but I have problem as I don't know how to multiple quote your message.

As for looking using a SQLite browser I prefer having timestamps on calibre. Much better !
Also I could agree into not having a single timestamp. I could identify the gap even in this scenario.

>True, but not finding a duplicate doesn't mean you don't have any duplicates,
>and I'm almost always looking for duplicates, not 99.99999999% duplicates.
>QTiny changes in the file mean that I wouldn't get a match with file hash/CRC.
>It's just not a tool I need. If I did need it, I'd use a third party file dupe finder.
> I'm not saying you don't need/want it.

In fact I am not whishing it as the perfect solution for any duplicate. It is the solution for situations where:

1) Calibre crashes in the middle of a batch adding.
2) You import a file which is phisically the same of a previous one.

In all other situations the standard tag based method is ok.

Believe me: today only I have had at least 10 crashes. I am going crazy without hashes.

Regards,
Giuseppe Chillemi

[speakingtohe]

Believe me: today only I have had at least 10 crashes. I am going crazy without hashes.

You can make your own hashes with other programs.

[Giuseppe Chillem]

Originally Posted by speakingtohe

@Giuseppe Chillemi
Can't you search for unknown?

How can this work? If you modify a book (even by importing to calibre) by adding metadata, changing title etc. the hash will change. The hash would have to be calculated before importing which would possibly slow things down on an overall level.

Right. The hash must be calculated at import before any modification of the tags.

For this request I have a precise scenario:
1) Frequent crashes during import
2) Frequent crashes of the tag reader without Calibre Crashes. This generates lots of "Unknown" by itself.
3) Tags not properly set which often led to "unkown" as result.

It is the kind of mess I am actually facing...

Last note: I am not a Phyton programmer... I work in REBOL.

Regards,
Giuseppe Chillemi

[Giuseppe Chillem]

Originally Posted by speakingtohe

You can make your own hashes with other programs.

Do you know what it means to generate hashes for a 70GB library at each crash of calibre ? It takes at least half an hour.

Regards,
Giuseppe Chillemi

[speakingtohe]

For this request I have a precise scenario:
1) Frequent crashes during import
2) Frequent crashes of the tag reader without Calibre Crashes. This generates lots of "Unknown" by itself.
3) Tags not properly set which often led to "unkown" as result.

Sorry I don't understand tag reader crashes. Can you explain?
I haven't had a calibre crash in months. Do you have sufficient hard drive space/memory

I am beginning to see why you are frustrated/upset but not why you are having these crashes.

Regards
Helen

[kovidgoyal]

Haven't I already told you to disable the PDF metadata reader to get rid of your crashes?

[speakingtohe]

Do you know what it means to generate hashes for a 70GB library at each crash of calibre ? It takes at least half an hour.

Yers I do which is why hash codes are not common practice AFAIK for dynamic database operations. 1/2 hour seems pretty fast to me even with i7.

I am thinking that has codes are not going to stop your crashes and that the crashes are the big problem.

[speakingtohe]

Haven't I already told you to disable the PDF metadata reader to get rid of your crashes?

There you go. Listen to kovid

[Giuseppe Chillem]

Originally Posted by speakingtohe

Sorry I don't understand tag reader crashes. Can you explain?
I haven't had a calibre crash in months. Do you have sufficient hard drive space/memory

I am beginning to see why you are frustrated/upset but not why you are having these crashes.

Regards
Helen

I have 8GB of RAM and 700GB HD space...
And Yes, I am FRUSTRATED :-)

About Tag Reader Crashes: they are different from Calibre crashes. When the tag reader hangs all the subsequent TAG read are "Unknown" until nex inport but calibre lives.

Regards,
Giuseppe Chillemi

[Giuseppe Chillem]

Originally Posted by kovidgoyal

Haven't I already told you to disable the PDF metadata reader to get rid of your crashes?

Yes you told me but the solution is unsuitable.

I am importing into calibre all the books collected during the years. If I disable the PDF Metadata reader I will lose COVERS too. Without covers I have to manually open each of 10.000 files to find NAME and AUTHOR or use the Metadata Editor and click 10.000 times to read the cover.

Also I want to dress your shoes and I think there is no way to know if the TAG reader is crashing or the cover reader is crashing as, I suppose, they belong to the same library.

As you gave me no chance about soliving the crashes issue (I think they depends from an external library you are using), the Hash solutions seems the only one suitable in my scenario.

Regards,
Giuseppe Chillemi