Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Closed Thread
 
Thread Tools Search this Thread
Old 09-14-2010, 10:49 AM   #316
GRiker
Comparer of the Ephemeris
GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.
 
Posts: 1,497
Karma: 424627
Join Date: Mar 2009
Device: iPad
There's a post by Leep in the Kindle forum describing his use of catalogs with the Kindle Collections feature. It's a clever way to generate a catalog specific to an author, then add it to a collection of that author's books on the Kindle.

G
GRiker is offline  
Old 09-17-2010, 03:50 AM   #317
alexxx
Connoisseur
alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.
 
Posts: 67
Karma: 479602
Join Date: Aug 2006
Device: Kindle DX
it is following that suggestion that I started enthusiastically creating single-author catalogs!
Then a bug hit me in the face - I guess it's a bug since I created many catalogs for other authors, before having this error, for J.G.Ballard's books:
Code:
ERROR: Conversion Error: <b>Failed</b>: Generate catalog

Generate catalog
Catalog_EPUB_MOBI(): Generating mobi for kindle_dx in GUI environment
 connected_device: 'Amazon Kindle'
 book count: 10
 Sections: Descriptions, Authors
 opts:
  authorClip: 100
  catalog_title: Ballard books
  connected_kindle: True
  descriptionClip: 380
  exclude_genre: .
  exclude_tags: ~,Catalog
  note_tag: *
  numbers_as_text: False
  read_tag: +
  search_text: None
  sort_by: None
  sort_descriptions_by_author: True
  sync: False
 Begin catalog source generation
     0 genre tags in database (excluding genres matching .):
      
     Genre summary: 0 active genre tags used in generating catalog with 10 titles
Traceback (most recent call last):
  File "/tmp/init.py", line 48, in <module>
  File "/home/kovid/build/calibre/src/calibre/utils/ipc/worker.py", line 99, in main
  File "/home/kovid/build/calibre/src/calibre/gui2/convert/gui_conversion.py", line 59, in gui_catalog
  File "/home/kovid/build/calibre/src/calibre/library/catalog.py", line 4641, in run
  File "/home/kovid/build/calibre/src/calibre/library/catalog.py", line 1231, in buildSources
  File "/home/kovid/build/calibre/src/calibre/library/catalog.py", line 3146, in generateNCXBySeries
IndexError: list index out of range

alessandro
alexxx is offline  
 
Advertisement
Old 09-17-2010, 06:43 AM   #318
alexxx
Connoisseur
alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.
 
Posts: 67
Karma: 479602
Join Date: Aug 2006
Device: Kindle DX
Another annotation/bug: half the time, creating catalogs for single authors (at a max of ~ 8-9 books each), the program is stuck, at job progress 100%, status:working.
I dont know what else to tell you, since no bug report is created - I can only kill the job.

alessandro
alexxx is offline  
Old 09-17-2010, 07:01 AM   #319
alexxx
Connoisseur
alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.alexxx ought to be getting tired of karma fortunes by now.
 
Posts: 67
Karma: 479602
Join Date: Aug 2006
Device: Kindle DX
big embarassment - sorry for the previous couple of posts:
everything has been solved by a simple calibre restart.


alessandro
alexxx is offline  
Old 09-17-2010, 08:26 AM   #320
GRiker
Comparer of the Ephemeris
GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.
 
Posts: 1,497
Karma: 424627
Join Date: Mar 2009
Device: iPad
@alexxx:
I believe that crassh you observed was caused by a bug when generating a catalog for a selection of books without any series info. That bug is fixed in the next release of calibre.

G
GRiker is offline  
Old 10-05-2010, 05:39 AM   #321
clockmaker
Member
clockmaker began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Jul 2010
Device: Kindle 2
Catalog Problem

Problem with Catalog in Calibre (Sorry about posting it in the normal forum first!)

I have tried about 5 times (with or without tags) to make a catalog (.mobi) of my rather large library; but without success. After about 2 hours of running I get the following error (only the last dozen lines given here):

Parsing file 'content/book_21442.html' as HTML
Forcing content/book_21442.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_21482.html' as HTML
Forcing content/book_21482.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_33595.html' as HTML
Forcing content/book_33595.html into XHTML namespace
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...

Exception <unknown>MemoryError: in <generator object gen at 0x75E2DF18> ignored
Exception <unknown>MemoryError: in <generator object __iter__ at 0x4E2F42B0> ignored
Python function terminated unexpectedly
(Error Code: 1)
sys.excepthook is missing
Traceback (most recent call last):
File "site-packages\calibre\ebooks\oeb\transforms\flatcss.py" , line 122, in __call__
File "site-packages\calibre\ebooks\oeb\stylizer.py", line 191, in __init__
<unknown>MemoryError

This is without the tags and with the tags it is a similar error

Initial parse failed:
Parsing file 'content/book_21442.html' as HTML
Forcing content/book_21442.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_3311.html' as HTML
Forcing content/book_3311.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_23694.html' as HTML
Forcing content/book_23694.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_3727.html' as HTML
Forcing content/book_3727.html into XHTML namespace
Initial parse failed:
Parsing file 'content/Genre_sciencefiction.html' as HTML
Forcing content/Genre_sciencefiction.html into XHTML namespace
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Python function terminated unexpectedly
(Error Code: 1)
Traceback (most recent call last):
File "site.py", line 103, in main
File "site.py", line 85, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 107, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 59, in gui_catalog
File "site-packages\calibre\library\catalog.py", line 4683, in run
File "site-packages\calibre\ebooks\conversion\plumber.py", line 915, in run
File "site-packages\calibre\ebooks\oeb\transforms\flatcss.py" , line 122, in __call__
File "site-packages\calibre\ebooks\oeb\transforms\flatcss.py" , line 147, in stylize_spine
File "site-packages\calibre\ebooks\oeb\stylizer.py", line 203, in __init__
File "site-packages\calibre\ebooks\oeb\stylizer.py", line 103, in __init__
File "site-packages\lxml\cssselect.py", line 523, in css_to_xpath
File "site-packages\lxml\cssselect.py", line 645, in parse
File "site-packages\lxml\cssselect.py", line 961, in __init__
MemoryError
Error in sys.exitfunc:
MemoryError


The database file is about 14 MB. Is there a limit on the size of the library? It appears that Catalog data is all used within the Ram. I am running it on 4 core (Intel 9300) machine with 4 GB of Ram. OR am I doing something wrong.

Thanks for any help.
clockmaker is offline  
Old 10-05-2010, 11:24 AM   #322
GRiker
Comparer of the Ephemeris
GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.
 
Posts: 1,497
Karma: 424627
Join Date: Mar 2009
Device: iPad
Quote:
Originally Posted by clockmaker View Post
Problem with Catalog in Calibre (Sorry about posting it in the normal forum first!)

I have tried about 5 times (with or without tags) to make a catalog (.mobi) of my rather large library; but without success. After about 2 hours of running I get the following error (only the last dozen lines given here):
...
MemoryError
Error in sys.exitfunc:
MemoryError

The database file is about 14 MB. Is there a limit on the size of the library? It appears that Catalog data is all used within the Ram. I am running it on 4 core (Intel 9300) machine with 4 GB of Ram. OR am I doing something wrong.

Thanks for any help.
A couple of clues to consider.

As @dwanthny stated in your original thread, it's not the size of your database, but the number of books in your database. If you're running for two hours and then running out of memory, I'm assuming that your database is pretty large, perhaps > 20,000 books?

If the system is running out of memory, this would be the first reported instance that I'm aware of. It's certainly possible. I recall some users early on describing catalog builds of ~8 hours, but they were successful. Check the earlier postings in this thread.

One thing that many users have done is to create smaller catalogs covering parts of their database. For example, you might have one called 'Reference Books'. You would filter the database before initiating the catalog build, usually based upon a tag set.

Try building some much smaller catalogs to make sure the problem doesn't lie elsewhere.

G
GRiker is offline  
Old 10-05-2010, 05:25 PM   #323
clockmaker
Member
clockmaker began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Jul 2010
Device: Kindle 2
Thank you dwanthny for your attention.

Version is 0.7.22
Number of books in the library: 32469
Authors: 8185
Series: 6795
Formats: 23
Tags: 6113
Publishers: 1576
News: 0
Formats: 23
Size of the metadata.db file: 31.052 Mega bytes

I have checked the first thread and I believe I am doing what all is recommended.
The error crops up at about the same place each time, whether tags are used or not. It is always memoryerror. It appears after more than 2 hours of processing and after the log message "Flattening CSS and remapping font sizes...". There is a fairly long time span (15 min. or so between this message and the error.

When it stops with error, the 4 GB memory is not fully used and the pagefile.sys 6.2 GB) is hardly touched. Commit in the task manager shows 4/10. The memory taken by the Calibre process is around 1.6 GB.

Does the catalog do all the catalog work within the ram without using the pagefile.sys? Is there any limit due to Python?

Is there a way to combine two Catalogs (with split and smaller databases) to one without going through the HTML route? Through the combining route, I am afraid the cross referencing in the catalog would be lost.
clockmaker is offline  
Old 10-05-2010, 05:32 PM   #324
clockmaker
Member
clockmaker began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Jul 2010
Device: Kindle 2
I have tried a database with about 100 books. It works fine without any problems. May be the problem is the large library.

Is there a way to combine two Catalogs (with split and smaller databases) to one without going through the HTML route? Through the combining route, I am afraid the cross referencing in the catalog would be lost.
clockmaker is offline  
Old 10-06-2010, 12:44 PM   #325
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,325
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@GRiker: you could make the processing processing more efficient by pre-flattening the CSS (i.e. using only class selectors and putting them all into a common css file that each individual HTML file links to)
kovidgoyal is offline  
Old 10-13-2010, 05:02 PM   #326
Tom SKP
Connoisseur
Tom SKP doesn't litterTom SKP doesn't litter
 
Posts: 68
Karma: 100
Join Date: Oct 2010
Device: PRS-350 & Nook Simple Touch with Glow light
More Book/Author Prefixs like the + Possible?

I'm a newbie and love exploring Calibre. It's great for printed books too!

The tag '+' for "Mark This Book as Read" is great.
Now is it possible to have other 'prefix' symbols available?

In addition to books I have and have not read, I have started listing some additional books that I do not yet have and would be able to code my Catalog for those too... with a 'o' prefix for example.

Is that possible?
Tom SKP is offline  
Old 10-13-2010, 05:11 PM   #327
GRiker
Comparer of the Ephemeris
GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.
 
Posts: 1,497
Karma: 424627
Join Date: Mar 2009
Device: iPad
There is one other special prefix in the current code, '*' for 'Note' if I remember correctly. You can use that however you want.

You can create an enhancement request as described in the first post in this thread. It would probably make more sense to be able to specify the contents of a custom column to be included in the generated catalog content, rather than using a special tag.

Another option would be to create a catalog of books you don't yet have in your library, and build a catalog of just those books. You could either filter on format:false or use a custom column.

G
GRiker is offline  
Old 10-13-2010, 05:29 PM   #328
Tom SKP
Connoisseur
Tom SKP doesn't litterTom SKP doesn't litter
 
Posts: 68
Karma: 100
Join Date: Oct 2010
Device: PRS-350 & Nook Simple Touch with Glow light
Quote:
Originally Posted by GRiker View Post
There is one other special prefix in the current code, '*' for 'Note' if I remember correctly. You can use that however you want.

You can create an enhancement request as described in the first post in this thread. It would probably make more sense to be able to specify the contents of a custom column to be included in the generated catalog content, rather than using a special tag.

Another option would be to create a catalog of books you don't yet have in your library, and build a catalog of just those books. You could either filter on format:false or use a custom column.

G
Thanks for the quick reply.
It's sooooo nice to look at a list under an author and see the + in the TOC.
Another symbol of my choosing would be great.
The simplest might be if the "Mark This Book as Read tag" field would allow multiple symbols (with comma separation) and display all of them in the TOCs.

Creating other catalogs or other tag sorts isn't as pretty or as functional. This is especially useful when shopping for hardcopy books. ie all information is in one place.

Tom
Tom SKP is offline  
Old 10-13-2010, 09:20 PM   #329
SingleEyePhotos
Member
SingleEyePhotos began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Sep 2010
Device: Kindle
I would very much like to make use of the catalog feature, but it's not giving me the option of any format that my Kindle will support. The initial post said that MOBI was an option, so long as the page setup settings were set to either Kindle or Kindle DX. I've set (and re-set) my output on the page setup as Kindle DX, but the only format options I get on the Catalog page are BIB, CSV, or XML. I've tried restarting Calibre, but that doesn't help. Using the current version, I go to Preferences=>Conversion=>Common Options=>Page Setup. Is that the correct path, as the instructions in this thread reference a much earlier version?

Thanks!
SingleEyePhotos is offline  
Old 10-14-2010, 09:48 AM   #330
GRiker
Comparer of the Ephemeris
GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.
 
Posts: 1,497
Karma: 424627
Join Date: Mar 2009
Device: iPad
Quote:
Originally Posted by SingleEyePhotos View Post
I would very much like to make use of the catalog feature, but it's not giving me the option of any format that my Kindle will support. The initial post said that MOBI was an option, so long as the page setup settings were set to either Kindle or Kindle DX. I've set (and re-set) my output on the page setup as Kindle DX, but the only format options I get on the Catalog page are BIB, CSV, or XML. I've tried restarting Calibre, but that doesn't help. Using the current version, I go to Preferences=>Conversion=>Common Options=>Page Setup. Is that the correct path, as the instructions in this thread reference a much earlier version?

Thanks!
You should be to select 'MOBI' in the Catalog format dropdown list.

If you're not seeing MOBI and EPUB, go to Preferences|Advanced|Plugins|Catalog generator plugins, confirm that Catalog_EPUB_MOBI is enabled.

G

Last edited by GRiker; 10-14-2010 at 09:51 AM.
GRiker is offline  
Closed Thread

Tags
book catalog

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Content Newspapers piling up in my Archived Items Millbrook Amazon Kindle 9 07-05-2012 04:09 PM
Creating Catalogs: Start here GRiker Library Management 0 01-26-2011 04:58 PM
Removing 'Archived Items' seagull Kindle Developer's Corner 7 09-29-2010 02:38 PM
Archived items question Corran006 Amazon Kindle 1 08-28-2010 03:52 AM
T?ol to create Structured catalogs? itimpi Calibre 5 11-19-2008 12:30 PM


All times are GMT -4. The time now is 07:30 AM.


MobileRead.com is a privately owned, operated and funded community.