Downloading TOC with metadata

TMSxMR · 07-07-2012, 06:01 AM

Hi All

As a previous "alfa book manager" user, I used that it downloads TOC for some books, this very useful to find specific subjects, but strangely calibre plugins don't do that (except one plugin I don't remember which one, and only for limited variety of books, and unfortunately merges it with comments!)

So my question why is that? amazon and Google provides TOC for lot of the books, is there a reason for that? or maybe some solution I don't know about?

kiwidude · 07-07-2012, 06:24 AM

B&N is the plugin that supports TOC downloading and yes it merges it into the comments.

As to the why - (1) there is no separate field for TOC in calibre, (2) it is not possible to download into a custom column from a metadata download plugin, and (3) I would suggest "most" users store fiction in their libraries, for which a TOC is irrelevant which is why it is not a "standard" field.

TMSxMR · 07-07-2012, 06:58 AM

But as i see calibre is quite flexible to target only fiction geeks, I have about 10 thousand books which are all scientific, and I know lot of people that have the same, and the metadata is available, and TOC can be easy stored in custom column (as I do now, but by hand), so I believe it's not quite justified to ignore that, if there is no other reasons for that.

If I knew python, no doubt i would modify plugins to include that, because i believe it's nothing more that parsing additional filed in the XML metadata that you grabbing from the providers.

kiwidude · 07-07-2012, 07:17 AM

As I said it is a current limitation in calibre that metadata plugins cannot download into custom columns. It isn't some grand conspiracy against non-fiction users, there are a number of ideas or things users would like to scrape using metadata download into a custom column. I only mentioned fiction as a reason why it is not a "standard" column.

There is a thread in the development forum which discusses this custom column download limitation, no-one has as yet picked up the ball to enhance calibre to allow it. I have no doubt it will happen "one day" but right now there is no-one actively working on it that I am aware of.

TMSxMR · 07-07-2012, 10:33 AM

Quote:

that metadata plugins cannot download into custom columns

Oh i missed that...
I will try to write some smal program in .Net to overcome that, maybe i will solve it...

kiwidude · 07-07-2012, 10:41 AM

Just to be clear, a program in .NET with enough work might get you around your particular problem allowing you to grab some TOC data but it won't really "solve" the bigger issue. Directly manipulating data in the calibre database as you would be doing from a .NET application is not exactly a supportable or recommended approach. Stuffing data into custom columns without using calibre's API to do so isn't something I would recommend doing but hey, its your database

The proper solution requires changes to calibre's plugin metadata merging logic and metadata download plugin api (which is all Python based), and then changes to the metadata download plugins to make use of the API changes.

TMSxMR · 07-08-2012, 07:17 AM

Actually I'm already manipulating my database heavily, same like i was doing with MS Access DB used by Alfa eBook Manager, and as I did in my Thumbnail browser for calibre database and seems that it doing the job without any conflicts.
Anyway I agree that it will be better to use calibre itself, but I don't know Python and have no time to learn it, so to overcome issues for example I'm deleting books from DB by connecting to calibre through command line instead of manipulating DB and files directly, i think the same can be done with metadata because there is also a command line for that, actually the really big issue is to grab metadata and TOC from the providers or from PDF files directly (not speaking of DJVU).

07-07-2012, 06:01 AM	#1
TMSxMR Member Posts: 11 Karma: 10 Join Date: Jul 2012 Device: Impression	Downloading TOC with metadata Hi All As a previous "alfa book manager" user, I used that it downloads TOC for some books, this very useful to find specific subjects, but strangely calibre plugins don't do that (except one plugin I don't remember which one, and only for limited variety of books, and unfortunately merges it with comments!) So my question why is that? amazon and Google provides TOC for lot of the books, is there a reason for that? or maybe some solution I don't know about?

07-07-2012, 06:58 AM	#3
TMSxMR Member Posts: 11 Karma: 10 Join Date: Jul 2012 Device: Impression	But as i see calibre is quite flexible to target only fiction geeks, I have about 10 thousand books which are all scientific, and I know lot of people that have the same, and the metadata is available, and TOC can be easy stored in custom column (as I do now, but by hand), so I believe it's not quite justified to ignore that, if there is no other reasons for that. If I knew python, no doubt i would modify plugins to include that, because i believe it's nothing more that parsing additional filed in the XML metadata that you grabbing from the providers. Last edited by TMSxMR; 07-07-2012 at 07:01 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Downloading ISBN metadata	apostrophistica	Plugins	2	02-25-2012 11:53 AM
Metadata Downloading in 0.8.27	LbrtyorDeath	Library Management	1	11-18-2011 07:04 PM
Metadata not downloading	Jay955528	Calibre	4	07-10-2011 02:26 AM
0.7.5.7 - Not Downloading Any Metadata	nynaevelan	Calibre	3	04-23-2011 11:38 AM
Downloading metadata not working	apastuszak	Library Management	4	02-20-2011 01:42 PM

07-07-2012, 06:24 AM	#2
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	B&N is the plugin that supports TOC downloading and yes it merges it into the comments. As to the why - (1) there is no separate field for TOC in calibre, (2) it is not possible to download into a custom column from a metadata download plugin, and (3) I would suggest "most" users store fiction in their libraries, for which a TOC is irrelevant which is why it is not a "standard" field.

07-07-2012, 07:17 AM	#4
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	As I said it is a current limitation in calibre that metadata plugins cannot download into custom columns. It isn't some grand conspiracy against non-fiction users, there are a number of ideas or things users would like to scrape using metadata download into a custom column. I only mentioned fiction as a reason why it is not a "standard" column. There is a thread in the development forum which discusses this custom column download limitation, no-one has as yet picked up the ball to enhance calibre to allow it. I have no doubt it will happen "one day" but right now there is no-one actively working on it that I am aware of.

07-07-2012, 10:41 AM	#6
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Just to be clear, a program in .NET with enough work might get you around your particular problem allowing you to grab some TOC data but it won't really "solve" the bigger issue. Directly manipulating data in the calibre database as you would be doing from a .NET application is not exactly a supportable or recommended approach. Stuffing data into custom columns without using calibre's API to do so isn't something I would recommend doing but hey, its your database The proper solution requires changes to calibre's plugin metadata merging logic and metadata download plugin api (which is all Python based), and then changes to the metadata download plugins to make use of the API changes.

07-08-2012, 07:17 AM	#7
TMSxMR Member Posts: 11 Karma: 10 Join Date: Jul 2012 Device: Impression	Actually I'm already manipulating my database heavily, same like i was doing with MS Access DB used by Alfa eBook Manager, and as I did in my Thumbnail browser for calibre database and seems that it doing the job without any conflicts. Anyway I agree that it will be better to use calibre itself, but I don't know Python and have no time to learn it, so to overcome issues for example I'm deleting books from DB by connecting to calibre through command line instead of manipulating DB and files directly, i think the same can be done with metadata because there is also a command line for that, actually the really big issue is to grab metadata and TOC from the providers or from PDF files directly (not speaking of DJVU).