Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 04-27-2011, 04:42 PM   #106
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,779
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
change author/title/delete/run a conversion.
kovidgoyal is online now   Reply With Quote
Old 04-29-2011, 04:48 PM   #107
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,606
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
v1.3 Released

Changes in this release:
  • Do all scanning as a background job to keep the UI responsive
  • Remove all interactive UI options - it will now always scan all formats in preferred order
  • Make sure that ISBN-13s start with 977, 978 or 979 (configurable).
  • Exclude the various repeating digit ISBNs of 1111111111 etc.
  • Exclude all html markup tags to prevent issues like the svg sizes being picked up as ISBNs
  • Include endash and other dash variants as possible separators
  • When scanning PDF documents, scan the last 5 pages in reverse order so it is the last ISBN found
  • Configuration option for ISBN13 prefixes and option to show updated books when extract completes
As has been mentioned several times in this thread - now that this version will run in the background as a Calibre job do not change the metadata or do a conversion for any of the books you selected to extract ISBN from.
kiwidude is offline   Reply With Quote
Advert
Old 05-05-2011, 04:18 PM   #108
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
A new sort of ISBN:

978-xxx-xxx-xxx^C

so a ^ just before the check digit.
Interesting one. Just seen once

ISBN-13: 978-0-451-46121^6 (alk. paper)
ISBN-10: 0-451-46121-5 (alk. paper)

(also interesting, the ISBN-10 is a 'normal' one
drMerry is offline   Reply With Quote
Old 05-05-2011, 04:25 PM   #109
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,606
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@drMerry - is that a retail book or a scan error? It does look very odd.
kiwidude is offline   Reply With Quote
Old 05-05-2011, 07:36 PM   #110
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
It IS odd.
It is a scan of a retail book.
I do not think it is an error, but I will look if I can find the a image of the book.

One other thing.
While you scan at background, there is a new problem.
I scanned 18000 (yes 18000) books at once.
No problem.
On details I could see a lot of numbers found.
@79% calibre crashed.
And no isbn was saved.
A fix to this would be to save direct (performance issue) or maybe to save it in a temp-file that you could look for next time you start calibre / the plugin.

But maybe I'm the only one with this problem.
drMerry is offline   Reply With Quote
Advert
Old 05-06-2011, 01:29 AM   #111
saddan
Junior Member
saddan began at the beginning.
 
Posts: 1
Karma: 10
Join Date: May 2011
Device: none
Thanks for this plugin!

I had problems with some books. One of them I would get this exception:

XMLSyntaxError: PCDATA invalid Char value 24, line 159, column 54

After some print statements, I noticed the xml generated in function _read_pdf_text from file scan.py had some invalid characters.

So I modified it to replace most of non-printable chars by something else ('_').

I'm attaching a diff of the modifications I did.

patch.scan.py.txt
saddan is offline   Reply With Quote
Old 05-06-2011, 05:31 AM   #112
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,606
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@Saddan - any chance you could PM me a link to the file that caused this error? I will delete it when done if it is commercial or whatever. I appreciate you taking the time to volunteer a fix, though there are some other functions in Calibre I could possibly use so if I could test with a file or two it would help greatly.

@drMerry - 18,000? Sigh. You wouldn't try to convert 18,000 books at once - Extract ISBN is doing a lot of the same steps underneath so I am not surprised it died. I cannot update ISBNs from the background as the database updates are single threaded, that is why it works the same way as metadata downloads in 0.8. Trying to "resume" a job after a Calibre crash is way more work than I can be bothered with to be honest, I would rather just not have it crash in the first place.

Did you see memory usage climbing before it crashed? The most likely explanation is a memory leak somewhere.
kiwidude is offline   Reply With Quote
Old 05-06-2011, 08:44 AM   #113
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,606
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
I've had a quick look at the "bulk extract" and there is a memory leak issue going on. However it is inside the Calibre code in the converters (non-pdf) as far as I can tell - certainly extracting from LRF files (which is horribly slow) the leak is pretty nasty and noticeable.

I'm sure at some point Kovid and co may take a look into this - in the meantime stick to extracting ISBN from small batches at a time and you will be fine. The other option I have is to run the extraction in separate worker processes like bulk conversions do. It would be a little slower probably but at least this problem should disappear and the GUI hopefully wouldn't choke every now and then like it does currently.
kiwidude is offline   Reply With Quote
Old 05-06-2011, 09:03 AM   #114
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
Quote:
Originally Posted by kiwidude View Post
@drMerry - 18,000? Sigh. You wouldn't try to convert 18,000 books at once - Extract ISBN is doing a lot of the same steps underneath so I am not surprised it died. I cannot update ISBNs from the background as the database updates are single threaded, that is why it works the same way as metadata downloads in 0.8. Trying to "resume" a job after a Calibre crash is way more work than I can be bothered with to be honest, I would rather just not have it crash in the first place.

Did you see memory usage climbing before it crashed? The most likely explanation is a memory leak somewhere.
I love to look at (over) the edge of possibilities.
And of-course you should want to convert / run it in small parts, but if I had, this leak would not be noticed at this point .
I did not check the mem, but I see you did it.

A (rather easy?) way to 'catch' this error (partly) would be to do the following:
Create a silent-run function. This function would run a scan and silently apply changes.
You than could use this function to create your own batch, and have a list of files todo whom you run in batches of 100, 1000, user-selected number of files at once. After each batch, apply changes, remove files from todo.
drMerry is offline   Reply With Quote
Old 05-06-2011, 10:59 AM   #115
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,606
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
v1.3.1 Released

Changes in this release:
  • Strip non-ascii characters from the pdfreflow xml which caused it to be invalid
  • Support the ^ character being part of the ISBN number
  • Attempt to minimise any memory leak issues caused by this plugin itself

Note that as per my post above there are still memory leak issues with some of the "conversions" that get run in the background. I've made sure that the plugin releases all file handles and resources that it creates asap so anything else is in the Calibre code. To try to work around that will require greater changes to this plugin than I want to make at this point but I will likely revisit it in future once I finish another new plugin that will work in a similar way.
kiwidude is offline   Reply With Quote
Old 05-06-2011, 11:22 AM   #116
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
hmm

error on install

Quote:
calibre, version 0.7.59
ERROR: Exceptie niet opgevangen: <b>OSError</b>:[Errno 2] No such file or directory

Traceback (most recent call last):
File "calibre_plugins.plugin_updater.dialogs", line 568, in _install_clicked
File "calibre_plugins.plugin_updater.dialogs", line 717, in _download_zip
File "site-packages\calibre\ptempfile.py", line 60, in __init__
File "tempfile.py", line 293, in mkstemp
File "tempfile.py", line 228, in _mkstemp_inner
OSError: [Errno 2] No such file or directory
drMerry is offline   Reply With Quote
Old 05-06-2011, 11:25 AM   #117
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,606
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Works fine for me... you sure you haven't run out of space or something with all your temp file issues? Try it again.
kiwidude is offline   Reply With Quote
Old 05-06-2011, 11:29 AM   #118
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
Quote:
Originally Posted by kiwidude View Post
Works fine for me... you sure you haven't run out of space or something with all your temp file issues? Try it again.
I'll keep trying, no success yet.
13.2 GB free, How big is this new version?
drMerry is offline   Reply With Quote
Old 05-06-2011, 11:38 AM   #119
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,606
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Do the usual checklist - is antivirus or something else blocking the downloads etc. Do other plugins install ok. Does it work if you install it manually etc.
kiwidude is offline   Reply With Quote
Old 05-06-2011, 11:43 AM   #120
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
On my second pc, I can now use the plugin.
I can see there is some mem-usage improvement

Thanks for this quick fix!

(first pc has no problem with other plugins, strange....)
EDIT:
Works
Don't know what the issue was..

Last edited by drMerry; 05-06-2011 at 12:06 PM.
drMerry is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Extract ISBN from PDF? mdroberts Calibre 14 12-16-2016 07:32 AM
[Old Thread] Extract ISBN from file name ChristianQ Calibre 59 12-09-2015 05:08 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM
[Old Thread] Auto Extract ISBN-Feature request UnraisedArc Calibre 60 03-23-2011 09:31 AM
Displaying ISBN column in the main GUI tilleydog Library Management 26 02-25-2011 04:08 AM


All times are GMT -4. The time now is 03:00 AM.


MobileRead.com is a privately owned, operated and funded community.