Creating Catalogs (Archived)

[GRiker]

There's a post by Leep in the Kindle forum describing his use of catalogs with the Kindle Collections feature. It's a clever way to generate a catalog specific to an author, then add it to a collection of that author's books on the Kindle.

G

[alexxx]

it is following that suggestion that I started enthusiastically creating single-author catalogs!
Then a bug hit me in the face - I guess it's a bug since I created many catalogs for other authors, before having this error, for J.G.Ballard's books:

ERROR: Conversion Error: <b>Failed</b>: Generate catalog

Generate catalog
Catalog_EPUB_MOBI(): Generating mobi for kindle_dx in GUI environment
 connected_device: 'Amazon Kindle'
 book count: 10
 Sections: Descriptions, Authors
 opts:
  authorClip: 100
  catalog_title: Ballard books
  connected_kindle: True
  descriptionClip: 380
  exclude_genre: .
  exclude_tags: ~,Catalog
  note_tag: *
  numbers_as_text: False
  read_tag: +
  search_text: None
  sort_by: None
  sort_descriptions_by_author: True
  sync: False
 Begin catalog source generation
     0 genre tags in database (excluding genres matching .):
      
     Genre summary: 0 active genre tags used in generating catalog with 10 titles
Traceback (most recent call last):
  File "/tmp/init.py", line 48, in <module>
  File "/home/kovid/build/calibre/src/calibre/utils/ipc/worker.py", line 99, in main
  File "/home/kovid/build/calibre/src/calibre/gui2/convert/gui_conversion.py", line 59, in gui_catalog
  File "/home/kovid/build/calibre/src/calibre/library/catalog.py", line 4641, in run
  File "/home/kovid/build/calibre/src/calibre/library/catalog.py", line 1231, in buildSources
  File "/home/kovid/build/calibre/src/calibre/library/catalog.py", line 3146, in generateNCXBySeries
IndexError: list index out of range

alessandro

[alexxx]

Another annotation/bug: half the time, creating catalogs for single authors (at a max of ~ 8-9 books each), the program is stuck, at job progress 100%, status:working.
I dont know what else to tell you, since no bug report is created - I can only kill the job.

alessandro

[alexxx]

big embarassment - sorry for the previous couple of posts:
everything has been solved by a simple calibre restart.


alessandro

[GRiker]

@alexxx:
I believe that crassh you observed was caused by a bug when generating a catalog for a selection of books without any series info. That bug is fixed in the next release of calibre.

G

[clockmaker]

Problem with Catalog in Calibre (Sorry about posting it in the normal forum first!)

I have tried about 5 times (with or without tags) to make a catalog (.mobi) of my rather large library; but without success. After about 2 hours of running I get the following error (only the last dozen lines given here):

Parsing file 'content/book_21442.html' as HTML
Forcing content/book_21442.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_21482.html' as HTML
Forcing content/book_21482.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_33595.html' as HTML
Forcing content/book_33595.html into XHTML namespace
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...

Exception <unknown>MemoryError: in <generator object gen at 0x75E2DF18> ignored
Exception <unknown>MemoryError: in <generator object __iter__ at 0x4E2F42B0> ignored
Python function terminated unexpectedly
(Error Code: 1)
sys.excepthook is missing
Traceback (most recent call last):
File "site-packages\calibre\ebooks\oeb\transforms\flatcss.py" , line 122, in __call__
File "site-packages\calibre\ebooks\oeb\stylizer.py", line 191, in __init__
<unknown>MemoryError

This is without the tags and with the tags it is a similar error

Initial parse failed:
Parsing file 'content/book_21442.html' as HTML
Forcing content/book_21442.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_3311.html' as HTML
Forcing content/book_3311.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_23694.html' as HTML
Forcing content/book_23694.html into XHTML namespace
Initial parse failed:
Parsing file 'content/book_3727.html' as HTML
Forcing content/book_3727.html into XHTML namespace
Initial parse failed:
Parsing file 'content/Genre_sciencefiction.html' as HTML
Forcing content/Genre_sciencefiction.html into XHTML namespace
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Python function terminated unexpectedly
(Error Code: 1)
Traceback (most recent call last):
File "site.py", line 103, in main
File "site.py", line 85, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 107, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 59, in gui_catalog
File "site-packages\calibre\library\catalog.py", line 4683, in run
File "site-packages\calibre\ebooks\conversion\plumber.py", line 915, in run
File "site-packages\calibre\ebooks\oeb\transforms\flatcss.py" , line 122, in __call__
File "site-packages\calibre\ebooks\oeb\transforms\flatcss.py" , line 147, in stylize_spine
File "site-packages\calibre\ebooks\oeb\stylizer.py", line 203, in __init__
File "site-packages\calibre\ebooks\oeb\stylizer.py", line 103, in __init__
File "site-packages\lxml\cssselect.py", line 523, in css_to_xpath
File "site-packages\lxml\cssselect.py", line 645, in parse
File "site-packages\lxml\cssselect.py", line 961, in __init__
MemoryError
Error in sys.exitfunc:
MemoryError


The database file is about 14 MB. Is there a limit on the size of the library? It appears that Catalog data is all used within the Ram. I am running it on 4 core (Intel 9300) machine with 4 GB of Ram. OR am I doing something wrong.

Thanks for any help.

[GRiker]

Originally Posted by clockmaker

Problem with Catalog in Calibre (Sorry about posting it in the normal forum first!)

I have tried about 5 times (with or without tags) to make a catalog (.mobi) of my rather large library; but without success. After about 2 hours of running I get the following error (only the last dozen lines given here):
...
MemoryError
Error in sys.exitfunc:
MemoryError

The database file is about 14 MB. Is there a limit on the size of the library? It appears that Catalog data is all used within the Ram. I am running it on 4 core (Intel 9300) machine with 4 GB of Ram. OR am I doing something wrong.

Thanks for any help.

A couple of clues to consider.

As @dwanthny stated in your original thread, it's not the size of your database, but the number of books in your database. If you're running for two hours and then running out of memory, I'm assuming that your database is pretty large, perhaps > 20,000 books?

If the system is running out of memory, this would be the first reported instance that I'm aware of. It's certainly possible. I recall some users early on describing catalog builds of ~8 hours, but they were successful. Check the earlier postings in this thread.

One thing that many users have done is to create smaller catalogs covering parts of their database. For example, you might have one called 'Reference Books'. You would filter the database before initiating the catalog build, usually based upon a tag set.

Try building some much smaller catalogs to make sure the problem doesn't lie elsewhere.

G

[clockmaker]

Thank you dwanthny for your attention.

Version is 0.7.22
Number of books in the library: 32469
Authors: 8185
Series: 6795
Formats: 23
Tags: 6113
Publishers: 1576
News: 0
Formats: 23
Size of the metadata.db file: 31.052 Mega bytes

I have checked the first thread and I believe I am doing what all is recommended.
The error crops up at about the same place each time, whether tags are used or not. It is always memoryerror. It appears after more than 2 hours of processing and after the log message "Flattening CSS and remapping font sizes...". There is a fairly long time span (15 min. or so between this message and the error.

When it stops with error, the 4 GB memory is not fully used and the pagefile.sys 6.2 GB) is hardly touched. Commit in the task manager shows 4/10. The memory taken by the Calibre process is around 1.6 GB.

Does the catalog do all the catalog work within the ram without using the pagefile.sys? Is there any limit due to Python?

Is there a way to combine two Catalogs (with split and smaller databases) to one without going through the HTML route? Through the combining route, I am afraid the cross referencing in the catalog would be lost.

[clockmaker]

I have tried a database with about 100 books. It works fine without any problems. May be the problem is the large library.

Is there a way to combine two Catalogs (with split and smaller databases) to one without going through the HTML route? Through the combining route, I am afraid the cross referencing in the catalog would be lost.

[kovidgoyal]

@GRiker: you could make the processing processing more efficient by pre-flattening the CSS (i.e. using only class selectors and putting them all into a common css file that each individual HTML file links to)

[Tom SKP]

I'm a newbie and love exploring Calibre. It's great for printed books too!

The tag '+' for "Mark This Book as Read" is great.
Now is it possible to have other 'prefix' symbols available?

In addition to books I have and have not read, I have started listing some additional books that I do not yet have and would be able to code my Catalog for those too... with a 'o' prefix for example.

Is that possible?

[GRiker]

There is one other special prefix in the current code, '*' for 'Note' if I remember correctly. You can use that however you want.

You can create an enhancement request as described in the first post in this thread. It would probably make more sense to be able to specify the contents of a custom column to be included in the generated catalog content, rather than using a special tag.

Another option would be to create a catalog of books you don't yet have in your library, and build a catalog of just those books. You could either filter on format:false or use a custom column.

G

[Tom SKP]

Originally Posted by GRiker

There is one other special prefix in the current code, '*' for 'Note' if I remember correctly. You can use that however you want.

You can create an enhancement request as described in the first post in this thread. It would probably make more sense to be able to specify the contents of a custom column to be included in the generated catalog content, rather than using a special tag.

Another option would be to create a catalog of books you don't yet have in your library, and build a catalog of just those books. You could either filter on format:false or use a custom column.

G

Thanks for the quick reply.
It's sooooo nice to look at a list under an author and see the + in the TOC.
Another symbol of my choosing would be great.
The simplest might be if the "Mark This Book as Read tag" field would allow multiple symbols (with comma separation) and display all of them in the TOCs.

Creating other catalogs or other tag sorts isn't as pretty or as functional. This is especially useful when shopping for hardcopy books. ie all information is in one place.

Tom

[SingleEyePhotos]

I would very much like to make use of the catalog feature, but it's not giving me the option of any format that my Kindle will support. The initial post said that MOBI was an option, so long as the page setup settings were set to either Kindle or Kindle DX. I've set (and re-set) my output on the page setup as Kindle DX, but the only format options I get on the Catalog page are BIB, CSV, or XML. I've tried restarting Calibre, but that doesn't help. Using the current version, I go to Preferences=>Conversion=>Common Options=>Page Setup. Is that the correct path, as the instructions in this thread reference a much earlier version?

Thanks!

[GRiker]

Originally Posted by SingleEyePhotos

I would very much like to make use of the catalog feature, but it's not giving me the option of any format that my Kindle will support. The initial post said that MOBI was an option, so long as the page setup settings were set to either Kindle or Kindle DX. I've set (and re-set) my output on the page setup as Kindle DX, but the only format options I get on the Catalog page are BIB, CSV, or XML. I've tried restarting Calibre, but that doesn't help. Using the current version, I go to Preferences=>Conversion=>Common Options=>Page Setup. Is that the correct path, as the instructions in this thread reference a much earlier version?

Thanks!

You should be to select 'MOBI' in the Catalog format dropdown list.

If you're not seeing MOBI and EPUB, go to Preferences|Advanced|Plugins|Catalog generator plugins, confirm that Catalog_EPUB_MOBI is enabled.

G