6.11 problem with adding large book collection - Page 2

kovidgoyal · 09-10-2009, 10:50 AM

It most definitely wont work with Python 3.

acekyoto · 09-10-2009, 05:36 PM

Well I guess I'll add books a few a time just when I'm reading to read them. I love the functionality of Calibre but I just can't figure this out. Does anyone else have their database in Calibre with more than 5000 books and have slowness adding more?

latexsalesman · 09-10-2009, 05:47 PM

I have a database of 28k+ and the adding is done at near the same speed as when it was much smaller. It is however much slower every time I do a search, move from my reader, and loading it up of course....

Checking the "Read metadata from the file name box" really has seemed to help me with my import speeds.

Good Luck!!!

I want to be your LatexSalesMan!!!!!

acekyoto · 09-11-2009, 12:17 AM

Thanks! I'll check that box when I get home and see if it helps.

acekyoto · 09-11-2009, 07:13 AM

Thanks...I checked that box but it doesn't seem to speed things up noticeably.

I did find out something. I found some of the books that were slowest to add were actually still in a winrar or zip archive. I guess the slowdown comes from calibre working within the archive...

RittTheRoo · 09-11-2009, 03:55 PM

I have been seeing the same thing as the original poster. Along with random crashes.
I'm running Win XP pro SP3. I add books using multiple books per directory option and read metadata from the file name setting. My calibre library is on a different hard drive than the source books, and it takes 3-4 seconds per book. I ran calibre using "calibre-debug -g" from the command prompt and I get a couple
"link hasn't been detected!"
Sometimes it will let me add the entire directory without crashing. When it does crash I get the following dialog box.
The instruction at "0x652c8d69" referenced memory at "0x00000008". The memory could not be "read"

The only other thing I get in the command prompt is a couple of these:
Job: 19 Read metadata (6 of 7) finished
Read metadata (6 of 7)
Job: 16 Read metadata (3 of 7) finished
Read metadata (3 of 7)

I've tried reinstalling Calibre and I'm not running Anti-virus or any other programs along with Calibre. Every book I'm trying to add is in PDF format.

I'm not sure what other information I can give you that may help diagnose the crashes.

kovidgoyal · 09-11-2009, 04:12 PM

PDF is a relatively slow format to import, this is because to read metadata from PDF, the entire file has to loaded in memory.

AS for the crashes, they only occur while adding books?

RittTheRoo · 09-11-2009, 04:39 PM

Yep, they only seem to happen when I'm adding books. I would have assumed that by checking the "read metadata only from file name" I wouldn't need to read the metadata from the PDF. My filename regular expression is:
(?P<author>[^_]+?) - (\[(?P<series>.*) (?P<series_index>[0-9]*)\] - )?(?P<title>.+)
Is there data that isn't in the filename that it still needs from the file itself?

kovidgoyal · 09-11-2009, 04:56 PM

No if you click read metadata only from file name then metadata wont be read from the file itself.

RittTheRoo · 09-11-2009, 05:35 PM

I'll run a memory diagnostic just to eliminate the possibility of bad RAM, but other than that I can't think of anything else to try.

*update*
RAM diagnostic checks fine.
I installed the latest version (6.12) of calibre and it seems more stable. I was able to add 4 directories without it crashing. It still takes 3-4 seconds to add each book but I can live with that. I might look into the command line to see if it will add books faster, and then use the GUI to edit metadata after.

I still don't understand why it has to load the entire PDF into memory to look at the metadata if I have the "Read metadata from filename" checked.

Tim Miller · 10-01-2009, 12:25 PM

I have a similar issue- have calibre on 3 PCs 1 xp laptop (dual core 4300)
1 Vista laptop (dual core 4200)
1 XP desktop (older p4 2.8)

earleir version very stable on vista laptop- had one 1.8 GB load that took 23 hours with no crash.
recently, adding thing over 50 MB will crash on all 3 PCs.
additionally, if I leave it running for more than 2 hours it crashes all by itself, with nothing loading

I have the windows error file if you want me to send it...

I love the program, it does everyhting I want, and manages my books and my reader 100 times better than stupid sony program, just need to fix crashes...

09-11-2009, 05:35 PM	#25
RittTheRoo Junior Member Posts: 6 Karma: 10 Join Date: Sep 2009 Device: Sony PRS-700, Nook	I'll run a memory diagnostic just to eliminate the possibility of bad RAM, but other than that I can't think of anything else to try. update RAM diagnostic checks fine. I installed the latest version (6.12) of calibre and it seems more stable. I was able to add 4 directories without it crashing. It still takes 3-4 seconds to add each book but I can live with that. I might look into the command line to see if it will add books faster, and then use the GUI to edit metadata after. I still don't understand why it has to load the entire PDF into memory to look at the metadata if I have the "Read metadata from filename" checked. Last edited by RittTheRoo; 09-11-2009 at 10:25 PM. Reason: Update

10-01-2009, 12:25 PM	#26
Tim Miller Junior Member Posts: 3 Karma: 10 Join Date: Jan 2009 Device: prs 505	calibre crashing I have a similar issue- have calibre on 3 PCs 1 xp laptop (dual core 4300) 1 Vista laptop (dual core 4200) 1 XP desktop (older p4 2.8) earleir version very stable on vista laptop- had one 1.8 GB load that took 23 hours with no crash. recently, adding thing over 50 MB will crash on all 3 PCs. additionally, if I leave it running for more than 2 hours it crashes all by itself, with nothing loading I have the windows error file if you want me to send it... I love the program, it does everyhting I want, and manages my books and my reader 100 times better than stupid sony program, just need to fix crashes...

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Tips for managing a large collection?	ficbot	Reading and Management	23	03-13-2010 02:11 AM
Adding books to a large database	Student1	Calibre	31	04-07-2009 06:43 PM
Large free ebook collection	tech_au	Deals and Resources (No Self-Promotion or Affiliate Links)	1	12-05-2008 08:47 AM
Problem with BookDesigner on large book	Megatron-UK	Sony Reader	6	04-07-2008 02:52 PM

09-10-2009, 10:50 AM	#16
kovidgoyal creator of calibre Posts: 43,875 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It most definitely wont work with Python 3.

09-10-2009, 05:36 PM	#17
acekyoto Member Posts: 12 Karma: 10 Join Date: Sep 2009 Device: PRS-600	Well I guess I'll add books a few a time just when I'm reading to read them. I love the functionality of Calibre but I just can't figure this out. Does anyone else have their database in Calibre with more than 5000 books and have slowness adding more?

09-10-2009, 05:47 PM	#18
latexsalesman Enthusiast Posts: 39 Karma: 124 Join Date: Nov 2008 Location: Lockport, NY Device: Pw2 @ Retina Mini	I have a database of 28k+ and the adding is done at near the same speed as when it was much smaller. It is however much slower every time I do a search, move from my reader, and loading it up of course.... Checking the "Read metadata from the file name box" really has seemed to help me with my import speeds. Good Luck!!! I want to be your LatexSalesMan!!!!!

09-11-2009, 12:17 AM	#19
acekyoto Member Posts: 12 Karma: 10 Join Date: Sep 2009 Device: PRS-600	Thanks! I'll check that box when I get home and see if it helps.

09-11-2009, 07:13 AM	#20
acekyoto Member Posts: 12 Karma: 10 Join Date: Sep 2009 Device: PRS-600	Thanks...I checked that box but it doesn't seem to speed things up noticeably. I did find out something. I found some of the books that were slowest to add were actually still in a winrar or zip archive. I guess the slowdown comes from calibre working within the archive...

09-11-2009, 03:55 PM	#21
RittTheRoo Junior Member Posts: 6 Karma: 10 Join Date: Sep 2009 Device: Sony PRS-700, Nook	I have been seeing the same thing as the original poster. Along with random crashes. I'm running Win XP pro SP3. I add books using multiple books per directory option and read metadata from the file name setting. My calibre library is on a different hard drive than the source books, and it takes 3-4 seconds per book. I ran calibre using "calibre-debug -g" from the command prompt and I get a couple "link hasn't been detected!" Sometimes it will let me add the entire directory without crashing. When it does crash I get the following dialog box. The instruction at "0x652c8d69" referenced memory at "0x00000008". The memory could not be "read" The only other thing I get in the command prompt is a couple of these: Job: 19 Read metadata (6 of 7) finished Read metadata (6 of 7) Job: 16 Read metadata (3 of 7) finished Read metadata (3 of 7) I've tried reinstalling Calibre and I'm not running Anti-virus or any other programs along with Calibre. Every book I'm trying to add is in PDF format. I'm not sure what other information I can give you that may help diagnose the crashes.

09-11-2009, 04:12 PM	#22
kovidgoyal creator of calibre Posts: 43,875 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	PDF is a relatively slow format to import, this is because to read metadata from PDF, the entire file has to loaded in memory. AS for the crashes, they only occur while adding books?

09-11-2009, 04:39 PM	#23
RittTheRoo Junior Member Posts: 6 Karma: 10 Join Date: Sep 2009 Device: Sony PRS-700, Nook	Yep, they only seem to happen when I'm adding books. I would have assumed that by checking the "read metadata only from file name" I wouldn't need to read the metadata from the PDF. My filename regular expression is: (?P<author>[^_]+?) - (\[(?P<series>.) (?P<series_index>[0-9])\] - )?(?P<title>.+) Is there data that isn't in the filename that it still needs from the file itself?

09-11-2009, 04:56 PM	#24
kovidgoyal creator of calibre Posts: 43,875 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No if you click read metadata only from file name then metadata wont be read from the file itself.

Advert

Advert