Calibre beta available that fully integrates custom fields

[Perkin]

chaley, In the first post on point four

For collections, putting #genre or #read into the collections field in device customization will do what you expect

you might want to add '(or any field's #lookupname)', or users might think only those two were added.

Thanks for all your work and help.

[itimpi]

I have just downloaded the latest beta and will be testing both my iPad and iPhone later today, so will feed back any issues I find.

[chaley]

Originally Posted by Sweetpea

On my WinCE/WinMobile devices, I use Mobipocket. ... So, I added my OPF files so that the "dc:title" is filled with (let's phrase it with Calibre names):

{author_sort} - {series} {series_index} - {title}

On my BBMini, ... (but show the filename instead).

The Sony does categories a bit more friendly ... So, I'd like to make my title more like

{series} {series_index} - {title}

...Wouldn't it be nice, if I were able to say what fields would be used to fill in those basic OPF fields (author and title, mostly)?

There be both dragons and opportunities here.

I think you aren't talking about an OPF that is separate from the book, but instead the metadata that is inside the book. Why do I say that?

If the reader used filesystem information, then you can already do what you want to do with the save template. This is your BBMini example. So you must be talking about something else.

I can't think of a single good reason a reader would look for an OPF file next to a book file. Why would it think it was there, and what would it do with it? So you are probably talking about metadata inside the books, especially epubs that have an OPF built in.

Assuming I have this right, here is a possibility. Calibre already updates the metadata for books when it sends them to devices or saves them to disks (call this send time). I could provide the notion of a 'metadata plugboard' that calibre could use at send time to 'connect' (semi-)arbitrary source metadata fields to (semi-)arbitrary destination fields. In other words, the metadata that calibre uses at send time to update the book would be what is left after applying the 'connections' in the plugboard.

Regarding your mobipocket example, I assume that it gets metadata from the book, not an OPF on the side. If true, then a plugboard that connects some composite column to title will do what you want. Freda (another reader) would do this, as it gets metadata from the book.

My understanding from reading kindle users' posts is that the kindle gets metadata from inside the .mobi file. If true, a plugboard that changes title and author_sort deals with the issues I know of. The first is the one you raise, getting series info into the title. The second relates to a kindle strangeness where authors must be separated by a ';' to sort properly.

Your sony example brings up a limitation. The plugboard will change the info inside the book, but does not change calibre's metadata for the book (this is a good thing). Devices with accessible databases (Sony, Kobo, and (I think) iDevices) use calibre's metadata to fill in the DB, not the book's, so the plugboards will not affect how calibre builds databases for these devices. Thus, plugboards can't change the displayed title as asked for in your sony example. All is not lost, however, because the file system path is easily seen on a book's info screen.

The reasons for database limitation relate to how metadata is exchanged between these devices and calibre. It goes back and forth several times during the cycle. Changing this to use plugboards both directions would be extremely complex, and isn't something I am willing to attempt.

Comments?

PS: given that I handle the sony driver, I could in fact build plugboard processing into the sony driver's database manipulation code. Two things bother me. The first is that the other DB-driven devices would behave differently, which is probably not a big deal. The second is that it would need to be absolutely right the first time, no corner cases, so that people who do manual management don't lose information.

[theducks]

Been using 904 on XP with my Astak PP, no uglies yet

[Sweetpea]

Originally Posted by chaley

There be both dragons and opportunities here.

<snip>

The reasons for database limitation relate to how metadata is exchanged between these devices and calibre. It goes back and forth several times during the cycle. Changing this to use plugboards both directions would be extremely complex, and isn't something I am willing to attempt.

Comments?

PS: given that I handle the sony driver, I could in fact build plugboard processing into the sony driver's database manipulation code. Two things bother me. The first is that the other DB-driven devices would behave differently, which is probably not a big deal. The second is that it would need to be absolutely right the first time, no corner cases, so that people who do manual management don't lose information.

Oh, I never talked about how to implement a scheme like this, it was just an idea I got when I started working with those fields... And how it would fit my way of working

You do raise some major objections, which I never thought off (but then, I've never looked into the code of Calibre...)

But a girl can dream, can't she? Besides, I know from personal experience, that a suggestion can have some far-reaching consequences as it will rattle around inside your brain and maybe trigger a brilliant idea.


Oh, the major problem I saw were circular references...

Create a custom colum "new_title"

{author} - {title}

and then use {#new_title} for your {title}...

[chaley]

Originally Posted by Sweetpea

Oh, I never talked about how to implement a scheme like this, it was just an idea I got when I started working with those fields... And how it would fit my way of working

Did the case that would work solve your problem (the MobiPocket reader case)?

But a girl can dream, can't she?

Isn't that illegal? Or immoral? Or something?

Oh, the major problem I saw were circular references...

Create a custom colum "new_title"

{author} - {title}

and then use {#new_title} for your {title}...

Arrgh. I took care of mutual recursion when displaying fields on the GUI, but I forgot to do it in save-to-disk. Fixed now.

Did you have the offending field displayed on the library view? And if so, didn't it display RECURSIVE_COMPOSITE_FIELD or some such in the table? I bet it did, and then you went ahead and used it anyway, just to see what would happen.

[Sweetpea]

Originally Posted by chaley

Did you have the offending field displayed on the library view? And if so, didn't it display RECURSIVE_COMPOSITE_FIELD or some such in the table? I bet it did, and then you went ahead and used it anyway, just to see what would happen.

Oh, in 904 it works, kind of... But no error messages. It only slows down Calibre something horrible.

Oh, and yes, that solution would work, at least for my Mobipocket books. If that was available, I could make my titles much more simple, and use Calibre's possibility of creating collections for my Sony.

[chaley]

Originally Posted by Sweetpea

Oh, in 904 it works, kind of... But no error messages. It only slows down Calibre something horrible.

The recursion test is extremely fast.

My guess is that the slowdown you are seeing comes from the auto-backup of your metadata. Although I am concerned about the 'horribly' part. I don't see that, but dwanthy reported something similar.

[Sweetpea]

Originally Posted by chaley

The recursion test is extremely fast.

My guess is that the slowdown you are seeing comes from the auto-backup of your metadata. Although I am concerned about the 'horribly' part. I don't see that, but dwanthy reported something similar.

It took me 5 minutes before I got the main screen somewhat responding again, so I could remove the offending recursing field out


I did notice another thing, though. I've been playing a lot with these custom fields, mostly in my save pattern.

I've added a custom field {#version}, which holds the current version of my books (as I always reformat them, I'll correct mistakes while reading and update the version). I decided to use that field in my save pattern:

{author_sort}/{author_sort} - {title} - v{#version:0>2s}

If I make a custom field ({#full_title} with this pattern I get:

Smith, Wilbur - Egyptian - 01 - River God - v00

However, when I use this as my save pattern, I get this:

Smith, Wilbur - Egyptian - 01 - River God - v_72.pdf

When I use the {#full_title} as save pattern, I get the same file name:

Smith, Wilbur - Egyptian - 01 - River God - v_72.pdf

Edit: oh, this is only true when {#version} = 0.

[kovidgoyal]

Does anyone else notice the auto backup slowing their calibre installs in 904? You can test it by going to preferences->advanced->misc and clicking backup. This will cause calibre to redo the backup for all books in your library.

[Sweetpea]

Originally Posted by kovidgoyal

Does anyone else notice the auto backup slowing their calibre installs in 904? You can test it by going to preferences->advanced->misc and clicking backup. This will cause calibre to redo the backup for all books in your library.

A bit sluggish, but nothing out of the ordinary and hardly noticeable. But then, I still have only a small library.

[kovidgoyal]

Originally Posted by Sweetpea

A bit sluggish, but nothing out of the ordinary and hardly noticeable. But then, I still have only a small library.

Size shouldn't matter, the backup happens at a constant rate irrespective of size.

[theducks]

Originally Posted by kovidgoyal

Does anyone else notice the auto backup slowing their calibre installs in 904? You can test it by going to preferences->advanced->misc and clicking backup. This will cause calibre to redo the backup for all books in your library.

If I could not hear the HD access, I would not know it is running.
1 to 2% bursts on CPU looking at Process explorer
1400 books

BTW should this show as a "Job" when running?

[theducks]

Bug report on 904

I Cancled Preferences while the backup was running.
I can not get focus back on the Main Calibre GUI (still running AFAICT), even to resize.

UPDATE
That was because it minimized over the XP start button, I just noticed it and it restored.
Which leads to anoter question
How do we know the task is done (other than keep retrying to really Cancel Preferences).

[kovidgoyal]

Originally Posted by theducks

BTW should this show as a "Job" when running?

No it's always running, automatically updating the OPF files when you edit metadata. If you don't edit anything, the thread sleeps so as not to consume resources.