Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 10-26-2010, 01:47 PM   #1
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Books disappear after calibre crash

Hello,
I have added some books to my library when the TAG reader crashed.
I have then read manually the image of the books whose name was "unknown" via "Modify Metadata Individually". After 8 books I have encountered the "problematic" pdf and calibre crashed.
After having realoaded calibre all the books I have edited before the problematic one disappeared. Where are they ?

As I encounter often this kind of problem I have a little request: could please make visible the TIME the book has been added to the library ? This would help to intercept and delete all the books added during the last run. Note, the time should be the same for all the books added in a single batch.

(Waiting for the crc32 + filesize duplicate detection...)

Giuseppe Chillemi

Last edited by Giuseppe Chillem; 10-26-2010 at 01:49 PM.
Giuseppe Chillem is offline   Reply With Quote
Old 10-26-2010, 02:00 PM   #2
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by Giuseppe Chillem View Post
After having realoaded calibre
Did you restart Calibre or actually have to reinstall it?

Quote:
all the books I have edited before the problematic one disappeared. Where are they ?
Did you look in the Library?

Quote:
As I encounter often this kind of problem I have a little request: could please make visible the TIME the book has been added to the library ? This would help to intercept and delete all the books added during the last run.
It appears you know this, but if not, the time is stored, but not shown. Sorting by date also sorts by time.

Quote:
Note, the time should be the same for all the books added in a single batch.
I disagree. I prefer to have it use the actual time added, which is what it does.

Quote:
(Waiting for the crc32 + filesize duplicate detection...)
I have considered working on metadata-based (author/title and isbn) duplicate detection, but not crc32 or filesize - neither of which is of interest to me. You already know you can do this with 3rd party tools.
Starson17 is offline   Reply With Quote
 
Enthusiast
Old 10-26-2010, 03:52 PM   #3
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
1) I have restart Calibre without reinstalling.

2) I have sorted the library by date and they were not there.

3) I know the time is there but I want to see it. It will solve once and for all the problem of multiple batch of books added the same day. In the actual scenario I can't recognize where one batch end and where another starts.

4) Can't understand why the precise adding time is better for you than the batch adding time (one and for all). The former doesn't give us any usefull information, the latter let us identify the latest batch.

5) File Hash + Duplicate tag is the complete solution for any problem. When a duplicate file hash is found you have 99.99999999% chance you have a duplicate. Using other tools complicate things too much as new files needs the duplicate library to be rebuild, even for one file added.

The final solution is:

File Hash duplicate check -> Ask for deletion.
Then:
Start the current Tag based duplicate check.

Berlieve me when I say your library will never have duplicates and believe me when I say this solves the Calibre Crash problem that generates duplicates. If the TAG library crashes and you have 100 "unknown - Unknown" books the tag mechanism won't work while the hash mechanism will find 100% of them, either checking against existing books or if you restart the batch, against new added books.

Regards
Giuseppe Chillemi
Giuseppe Chillem is offline   Reply With Quote
Old 10-26-2010, 04:16 PM   #4
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by Giuseppe Chillem View Post
1) I have restart Calibre without reinstalling.
Good. I've never needed to reinstall (and I can't recall any crashes either, not that I disbelieve you.)

Quote:
2) I have sorted the library by date and they were not there.
I was suggesting you use your OS file browser to look directly in the folder set as the Calibre Library folder. It should have your books. I have no idea why they don't show up in Calibre.

Quote:
3) I know the time is there but I want to see it.
I understood that.
Quote:
It will solve once and for all the problem of multiple batch of books added the same day. In the actual scenario I can't recognize where one batch end and where another starts.
I have no trouble identifying the breaks. The gap in time is much larger. I view this info in an SQL Browser when I want it.

Quote:
4) Can't understand why the precise adding time is better for you than the batch adding time (one and for all). The former doesn't give us any usefull information, the latter let us identify the latest batch.
I don't see how you can identify the batch if you don't see the time. I can identify both, but I have to use the SQL browser. I've occasionally needed to know how long it takes to add files. I can get that from the exact time during a batch add.
Quote:
5) File Hash + Duplicate tag is the complete solution for any problem. When a duplicate file hash is found you have 99.99999999% chance you have a duplicate.
True, but not finding a duplicate doesn't mean you don't have any duplicates, and I'm almost always looking for duplicates, not 99.99999999% duplicates. Tiny changes in the file mean that I wouldn't get a match with file hash/CRC. It's just not a tool I need. If I did need it, I'd use a third party file dupe finder. I'm not saying you don't need/want it.

Quote:
The final solution is:

File Hash duplicate check -> Ask for deletion.
I keep certain 100% duplicates I don't want deleted. Specifically, If I want 2 different copies of a particular format, I usually copy the record, with all formats, then change only the one format.

Quote:
believe me when I say this solves the Calibre Crash problem that generates duplicates.
I've never had a duplicate due to a "Calibre Crash."

Quote:
If the TAG library crashes and you have 100 "unknown - Unknown" books the tag mechanism won't work while the hash mechanism will find 100% of them, either checking against existing books or if you restart the batch, against new added books.
I don't fully understand this. If you are getting a crash using tags, you should report it. I've never seen one. I have lots of "Unknown" authors, but they are truly unknown and they aren't duplicates very often. IIRC from your earlier posts, you are seeing "Unknown" because you have metadata pulled from inside the file, and the internal metadata isn't there, or it's poor. I can understand in that case that you want hash-based duplicate detection, as your metadata is so poor, but it's just not something that I would ever use. My books almost always have some identifying author or title information in the filename to work with.
Starson17 is offline   Reply With Quote
Old 10-26-2010, 04:29 PM   #5
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,596
Karma: 25170848
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
@Giuseppe Chillemi
Can't you search for unknown?

Quote:
File Hash duplicate check -> Ask for deletion.
Then:
Start the current Tag based duplicate check.

Berlieve me when I say your library will never have duplicates and believe me when I say this solves the Calibre Crash problem that generates duplicates. If the TAG library crashes and you have 100 "unknown - Unknown" books the tag mechanism won't work while the hash mechanism will find 100% of them, either checking against existing books or if you restart the batch, against new added books.
How can this work? If you modify a book (even by importing to calibre) by adding metadata, changing title etc. the hash will change. The hash would have to be calculated before importing which would possibly slow things down on an overall level.

Even a hash based on title only will be useless as many titles seem to be chosen by different authors. Title-author, you have to deal with things like spaces after initials or periods after initials.

For example J. D. Robb could be written in many ways
J. D. Robb
J.D. Robb
J D Robb
Norah Roberts as J. D. Robb.

Only hash code that would actually work is one generated on original file and this could not be done until file was added.


While pretty old school, hash codes still have their place, but I don't think it is a viable solution in this instance. The current duplicate detection is pretty good although not foolproof.

Of course you can prove me wrong by writing a foolproof hash code based calibre module to do this

Regards
Helen
speakingtohe is offline   Reply With Quote
Old 10-26-2010, 05:18 PM   #6
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Sorry17 Starson but I have problem as I don't know how to multiple quote your message.

As for looking using a SQLite browser I prefer having timestamps on calibre. Much better !
Also I could agree into not having a single timestamp. I could identify the gap even in this scenario.

>True, but not finding a duplicate doesn't mean you don't have any duplicates,
>and I'm almost always looking for duplicates, not 99.99999999% duplicates.
>QTiny changes in the file mean that I wouldn't get a match with file hash/CRC.
>It's just not a tool I need. If I did need it, I'd use a third party file dupe finder.
> I'm not saying you don't need/want it.

In fact I am not whishing it as the perfect solution for any duplicate. It is the solution for situations where:

1) Calibre crashes in the middle of a batch adding.
2) You import a file which is phisically the same of a previous one.

In all other situations the standard tag based method is ok.

Believe me: today only I have had at least 10 crashes. I am going crazy without hashes.

Regards,
Giuseppe Chillemi
Giuseppe Chillem is offline   Reply With Quote
Old 10-26-2010, 05:22 PM   #7
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,596
Karma: 25170848
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Believe me: today only I have had at least 10 crashes. I am going crazy without hashes.
You can make your own hashes with other programs.
speakingtohe is offline   Reply With Quote
Old 10-26-2010, 05:23 PM   #8
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Quote:
Originally Posted by speakingtohe View Post
@Giuseppe Chillemi
Can't you search for unknown?

How can this work? If you modify a book (even by importing to calibre) by adding metadata, changing title etc. the hash will change. The hash would have to be calculated before importing which would possibly slow things down on an overall level.
Right. The hash must be calculated at import before any modification of the tags.

For this request I have a precise scenario:
1) Frequent crashes during import
2) Frequent crashes of the tag reader without Calibre Crashes. This generates lots of "Unknown" by itself.
3) Tags not properly set which often led to "unkown" as result.

It is the kind of mess I am actually facing...

Last note: I am not a Phyton programmer... I work in REBOL.

Regards,
Giuseppe Chillemi
Giuseppe Chillem is offline   Reply With Quote
Old 10-26-2010, 05:25 PM   #9
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Quote:
Originally Posted by speakingtohe View Post
You can make your own hashes with other programs.
Do you know what it means to generate hashes for a 70GB library at each crash of calibre ? It takes at least half an hour.

Regards,
Giuseppe Chillemi
Giuseppe Chillem is offline   Reply With Quote
Old 10-26-2010, 05:29 PM   #10
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,596
Karma: 25170848
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
For this request I have a precise scenario:
1) Frequent crashes during import
2) Frequent crashes of the tag reader without Calibre Crashes. This generates lots of "Unknown" by itself.
3) Tags not properly set which often led to "unkown" as result.
Sorry I don't understand tag reader crashes. Can you explain?
I haven't had a calibre crash in months. Do you have sufficient hard drive space/memory

I am beginning to see why you are frustrated/upset but not why you are having these crashes.

Regards
Helen
speakingtohe is offline   Reply With Quote
Old 10-26-2010, 05:33 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,915
Karma: 5035037
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Haven't I already told you to disable the PDF metadata reader to get rid of your crashes?
kovidgoyal is online now   Reply With Quote
Old 10-26-2010, 05:36 PM   #12
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,596
Karma: 25170848
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Do you know what it means to generate hashes for a 70GB library at each crash of calibre ? It takes at least half an hour.
Yers I do which is why hash codes are not common practice AFAIK for dynamic database operations. 1/2 hour seems pretty fast to me even with i7.

I am thinking that has codes are not going to stop your crashes and that the crashes are the big problem.
speakingtohe is offline   Reply With Quote
Old 10-26-2010, 05:37 PM   #13
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,596
Karma: 25170848
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Haven't I already told you to disable the PDF metadata reader to get rid of your crashes?
There you go. Listen to kovid
speakingtohe is offline   Reply With Quote
Old 10-26-2010, 07:19 PM   #14
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Quote:
Originally Posted by speakingtohe View Post
Sorry I don't understand tag reader crashes. Can you explain?
I haven't had a calibre crash in months. Do you have sufficient hard drive space/memory

I am beginning to see why you are frustrated/upset but not why you are having these crashes.

Regards
Helen
I have 8GB of RAM and 700GB HD space...
And Yes, I am FRUSTRATED :-)

About Tag Reader Crashes: they are different from Calibre crashes. When the tag reader hangs all the subsequent TAG read are "Unknown" until nex inport but calibre lives.

Regards,
Giuseppe Chillemi
Giuseppe Chillem is offline   Reply With Quote
Old 10-26-2010, 07:28 PM   #15
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Quote:
Originally Posted by kovidgoyal View Post
Haven't I already told you to disable the PDF metadata reader to get rid of your crashes?
Yes you told me but the solution is unsuitable.

I am importing into calibre all the books collected during the years. If I disable the PDF Metadata reader I will lose COVERS too. Without covers I have to manually open each of 10.000 files to find NAME and AUTHOR or use the Metadata Editor and click 10.000 times to read the cover.

Also I want to dress your shoes and I think there is no way to know if the TAG reader is crashing or the cover reader is crashing as, I suppose, they belong to the same library.

As you gave me no chance about soliving the crashes issue (I think they depends from an external library you are using), the Hash solutions seems the only one suitable in my scenario.

Regards,
Giuseppe Chillemi
Giuseppe Chillem is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Kobo Books Disappear Themus Kobo Reader 6 10-09-2010 09:19 AM
Preferences causes Calibre to crash PKFFW Calibre 2 06-07-2010 06:01 PM
Calibre 0.6.51 Preferences crash GexMax Calibre 0 05-02-2010 09:04 AM
Ebook DRM provider goes dark, the books you paid for disappear orwell2k News 17 02-04-2010 03:23 PM
Conversions crash calibre slantybard Calibre 0 07-20-2009 12:50 AM


All times are GMT -4. The time now is 04:07 AM.


MobileRead.com is a privately owned, operated and funded community.