Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 09-19-2010, 06:12 PM   #1
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,920
Karma: 1673776
Join Date: Jan 2010
Location: France
Device: Many android devices
Calibre beta available that fully integrates custom fields

Edit: New information is underlined.

Dear all,

This beta makes a significant upgrade of calibre available for testing. This note describes beta release 0.7.906.

The major enhancements (that I remember) are:
  • Custom fields are usable in save-to-disk and send-to-device templates, and in collections. Template items like {#mycolumn} will substitute the value of the custom field into the template.
  • Various functions are available in templates, used for constructing context-sensitive folder paths. See the templates tutorial for details.
  • There is a new custom field type, ‘column built from other columns’ (composite columns). This type lets you construct *read-only* columns containing any of the available metadata fields, using the template syntax. For example, you can construct a column showing the ISBN by creating a composite column containing the template {isbn}. Fields like {format} and {author_sort} also work. You can even do ‘{tags:10.10}-{series}’ to build a column that contains the first 10 characters of tags, followed by a hyphen then series. These fields can be very useful in templates.
  • A new feature, plugboards, is now available for controlling what metadata is written into books during send-to-device and save-to-disk operations. A plugboard permits you to specify a template (which can, of course, use composite columns) to provide the data to write into the book's metadata. You can modify the following fields: authors, author_sort, language, publisher, tags, title, title_sort. This feature should help those of you who want to use different metadata in your books on devices to solve sorting or display issues. Please note: this feature does not affect calibre's metadata, nor does it affect what is written into a Sony's database.
  • For collections, putting #genre or #read (or any field's #lookupname) into the collections field in device customization will do what you expect. A tweak, sony_collection_renaming_rules, can be used to control how collection names appear on the Sony.
  • Custom fields are displayed in the content server and in book information. The standard content server display shows long fields (e.g., lots of tags) in an elided form. Full information is available by clicking on the book. A tweak has been added to select which custom fields to, or not to, display.
  • OPFs now contain custom field information.
  • The calibre device cache contains custom field information.
  • Metadata is automatically backed up to an OPF in the book directory. The first time you run a beta release, OPFs will be created for all your books at the rate of around 30 per minute. You can see how many books' metadata remains to be backed up by looking at the command 'Library Backup Status', available on the library pull-down. You can force a complete rewrite of metadata using the button on preferences -> advanced -> miscellaneous.
  • Calibre now has a database recover CLI command 'calibredb restore_database'. This command walks over a library gathering all the OPF files it finds (the ones made by the backup), then rebuilds the library database from that information. Custom columns are restored. Saved searches, user-defined categories, and saved conversion options are not restored. Restoration takes around 1 second per book.
  • Search & replace that uses/operates upon custom fields, can copy from one field to another, and can change the case of text. The command has two modes: text and regular expression. The text mode is easiest for simple replacements or when just changing case.
  • Bulk edit can now force series-type columns to empty, and can force renumbering to start at some value.
  • tweak-epub. See the forum post http://www.mobileread.com/forums/showthread.php?t=99875.

The beta manual is available here.

This is a near-final beta release. We expect that the release will be final this week.

There haven’t been any changes to the database schema, which means that it is possible to try the beta, then uninstall and revert to a production release (an uninstall is necessary). I add the ‘possible’, because you will need to manually undo any changes to preferences (e.g., custom fields in templates), and the production releases will not like composite columns. The best approach is to copy your library, install the beta, then try a few things to see if it works well enough in your configuration. If it does not, then uninstall calibre, reinstall the production version, and switch back to the original library.

Please post comments on this thread.

I will edit this post as necessary to keep it up-to-date.

The beta is available at http://calibre-ebook.com/downloads/betas/. The .msi is for windows, the .dmg is for macs, and the other two are for linux.

Last edited by chaley; 09-30-2010 at 04:34 AM. Reason: Latest feature list
chaley is offline   Reply With Quote
Old 09-19-2010, 07:20 PM   #2
Leep
Guru
Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.Leep ought to be getting tired of karma fortunes by now.
 
Leep's Avatar
 
Posts: 739
Karma: 1018859
Join Date: Jun 2008
Location: Near Seattle
Device: kindle1, K3, K3G (thanks MR), iTouch, Kindle Touch
I'm sure I speak for all in extending a very big thank you to you and Kovid. Calibre is great and keeps getting better.

cheers
Leep is offline   Reply With Quote
Old 09-20-2010, 01:03 AM   #3
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,287
Karma: 6022733
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Trying to save Save to disk preferences (changed only a tick box)


Code:
ERROR: Invalid template: <p>The template 00NoSeries/{author_sort[0]}_authors/{authors}/{title}_{id}-{authors} is invalid:<br>string index out of range
I have been using this mask for months
theducks is offline   Reply With Quote
Old 09-20-2010, 04:30 AM   #4
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,920
Karma: 1673776
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by theducks View Post
Trying to save Save to disk preferences (changed only a tick box)
Thanks. Fixed.

The problem is that the original validation code required foreknowledge of all possible fields, which isn't possible any more. I originally removed validation, but recently I thought I had a solution. Clearly I didn't.

The same problem would arise when using save paths. For example, if you used {series[0]} and the series was null, an exception would (well, will in the current beta) be thrown. I fixed that too.
chaley is offline   Reply With Quote
Old 09-20-2010, 10:22 PM   #5
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
I've misplaced the bazaar launchpad link that Kovid sent out for source on this. Can someone post it? Thanks!
Starson17 is offline   Reply With Quote
Old 09-20-2010, 11:24 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,450
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
lp:~kovid/calibre/custom-serializer
kovidgoyal is offline   Reply With Quote
Old 09-21-2010, 09:58 AM   #7
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kovidgoyal View Post
lp:~kovid/calibre/custom-serializer
Thanks!
Starson17 is offline   Reply With Quote
Old 09-21-2010, 12:27 PM   #8
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,920
Karma: 1673776
Join Date: Jan 2010
Location: France
Device: Many android devices
For developers

Notes for those of you who do development from time to time:

The new metadata interface is much easier and more convenient to use than the older interfaces. Given a book, you can get values and column metadata. There is a method that formats the value for you, and another method that accepts an arbitrary template, returning the resulting string.

If you are working in database-land, then use db.get_metadata to get a book structure. Get_metadata is now cached, so the performance is acceptable. The book is a Metadata instance (ebooks/metadata/book/base.py), so all the methods in that class will be available to you. You can also use a new set of field-oriented interface methods to get metatada, to get lists of field keys (sort/search names), and to get field values by name.

If you are working in device-driver land, then the book you are used to is now a Metadata instance. The JSON cache now contains all the attributes, so you can trust the book to have whatever standard or custom attributes were there when the book was last written to the cache.

My experience is that Metadata (book) instances are much easier to use than what we had before. Getting a field is as easy as book.get('field'). Getting a field's type is book.metadata_for_field('field')['datatype']. Getting a formatted value is book.format_field('field').

Don't hesitate to ask questions. Even some suggestions will be entertained.
chaley is offline   Reply With Quote
Old 09-21-2010, 12:42 PM   #9
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by chaley View Post
Notes for those of you who do development from time to time
I can see why the rewrite was such a big effort. Behind the scenes lots of code to access the database had to be rewritten to use the new metadata interface. Am I correct in thinking that this is part of the long term goal to support other types of library structures, beyond the current /drive/folder/file system?

If it's of any help, one more datapoint for you is: My first tests of saving files, using custom metadata to name the saved files, checking the custom metadata in .opf files and reimporting files with the custom metadata going back into the correct custom column have all been successful.
Starson17 is offline   Reply With Quote
Old 09-21-2010, 01:32 PM   #10
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,920
Karma: 1673776
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by Starson17 View Post
I can see why the rewrite was such a big effort. Behind the scenes lots of code to access the database had to be rewritten to use the new metadata interface. Am I correct in thinking that this is part of the long term goal to support other types of library structures, beyond the current /drive/folder/file system?
Yes.

This project got started when I and Kovid started discussing how to redo the database/library interface to be general. We (he, really) quickly realized that we needed to find a way to do it stepwise. The idea that presented itself was to do an interface that incorporated custom fields, then be able to serialize that information to OPFs and calibre's cache. That is where we are now.

We (I) learned a great deal during this process. One example outcome was the move to table-driven data types (field_metadata). This allowed us to settle on a 'key' attribute naming structure, where the key is the one-true field name. This led to the attribute-level interface in Metadata. And so on.

The next step - refactoring of the library to use arbitrary storage, will probably wait a while. First, Kovid will drive that project, which is reasonable because he will need to live with it, and he is busy. Second, we probably should slow down the change rate of 'fundamental stuff', primarily so things don't get broken out of ignorance.
Quote:
If it's of any help, one more datapoint for you is: My first tests of saving files, using custom metadata to name the saved files, checking the custom metadata in .opf files and reimporting files with the custom metadata going back into the correct custom column have all been successful.
That is very good to hear.
chaley is offline   Reply With Quote
Old 09-21-2010, 02:46 PM   #11
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by chaley View Post
The idea that presented itself was to do an interface that incorporated custom fields, then be able to serialize that information to OPFs and calibre's cache. That is where we are now.
And this explains why this code branch is called "custom-serializer." I did wonder a bit.
Starson17 is offline   Reply With Quote
Old 09-21-2010, 05:48 PM   #12
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,920
Karma: 1673776
Join Date: Jan 2010
Location: France
Device: Many android devices
Near-final beta binary now released

All,

Version 0.7.901 of the beta has been uploaded to the server. This version:
  1. fixes the only reported bug (thanks, theducks).
  2. incorporates changes to search/replace, providing a text-only (non-RE) interface and the ability to copy one field to another.
  3. incorporates changes to bulk edit, permitting renumbering a series starting at an arbitrary integer.
  4. includes a performance improvement, avoiding disk accesses when the same book is accessed in multiple places.
  5. fixes bug in content server display, where tag expansion included some extra characters.
  6. contains several fixes that should reduce problems with 'access error' exceptions. (Will also be in the next standard 0.7 release.)
  7. permits you to 'tweak' an epub by exploding it into its component parts, changing things, then rebuilding the epub. (Will also be in the next standard 0.7 release.)
  8. all other fixes made to date to the 0.7 release series.

There is nothing in the 0.7 standard releases that is not included in this beta, and will not be included in the production release when it happens.

This could easily be the last beta before this release series inserts itself into the existing 0.7 sequence. Our target is to switch over next week, when this beta will become 0.7.21.

If you have any concerns, please test this release. If we don't know about a problem, we can't fix it.

Note: if you install the beta, you will need to uninstall it before you will be able successfully to install the subsequent 0.7 releases.
chaley is offline   Reply With Quote
Old 09-21-2010, 10:51 PM   #13
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,287
Karma: 6022733
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Don't kill me ;)

Code:
---------------------------
ERROR: ERROR: Unhandled exception
---------------------------
<b>AttributeError</b>:'NoneType' object has no attribute 'split'
---------------------------
OK   Copied   Hide Details...   
---------------------------
901,
XP
HanlinV5 (Astak, Lbook FW)
Happens a short while after I connect the device and it is/has fetching metadata.calibre to show titles on device
I uninstalled 7.19 before installing 7.901

Update. I watched the Task list
Getting list of books on device: approaches 100% @ 55secs when it fails.

Last edited by theducks; 09-21-2010 at 10:56 PM. Reason: updated
theducks is offline   Reply With Quote
Old 09-22-2010, 03:12 AM   #14
N13L5
tenjooberrymuds
N13L5 began at the beginning.
 
Posts: 58
Karma: 12
Join Date: Sep 2010
Device: Android
How portable is a windows installation of Calibre, do you go by "microsoft best err.. worst practice" and write everything to the registry, or can I install the beta copy of Calibre on another drive to mess around on, without killing my first installation?
N13L5 is offline   Reply With Quote
Old 09-22-2010, 05:32 AM   #15
Manichean
Wizard
Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!
 
Manichean's Avatar
 
Posts: 3,130
Karma: 80520
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
I'm getting the same error theducks is getting. 0.7.901, XP SP3, Cybook Gen3. Reproducible, occurs everytime the device is plugged in and Calibre tries to get metadata (at least, that's what it says in the jobs manager).
Full error report:
ERROR: FEHLER: Unbehandelte Ausnahme: <b>AttributeError</b>:'NoneType' object has no attribute 'split'

Traceback (most recent call last):
File "site-packages\calibre\gui2\__init__.py", line 297, in dispatch
File "site-packages\calibre\gui2\device.py", line 753, in metadata_downloaded
File "site-packages\calibre\gui2\device.py", line 1384, in set_books_in_library
File "site-packages\calibre\library\database2.py", line 593, in get_metadata
AttributeError: 'NoneType' object has no attribute 'split'

Edited to add: Should this matter, I only made a copy of my library on another computer and tested that, so metadata on the device should be the same as in the library.
Still further edited to add: After messing around with the amazing (Really! I love it to death!) S&R- function, I discovered that the last point obviously doesn't matter. Same error, different metadata. Also, I can't access any of the books on the device, and the "On device"- column isn't created. The buttons in the main toolbar for reader memory and expansion card appear correctly, though.
Next edit: This error also occurs when trying to connect to a folder (I wanted to try something without messing up the books on my device). Also, when trying to send books to said folder, I get another error, altough that may be a consequence of the first one:
ERROR: FEHLER: Unbehandelte Ausnahme: <b>AttributeError</b>:'list' object has no attribute 'add_book'

Traceback (most recent call last):
File "site-packages\calibre\gui2\__init__.py", line 297, in dispatch
File "site-packages\calibre\gui2\device.py", line 1287, in books_uploaded
File "site-packages\calibre\gui2\device.py", line 349, in add_books_to_metadata
File "site-packages\calibre\devices\usbms\driver.py", line 246, in add_books_to_metadata
AttributeError: 'list' object has no attribute 'add_book'

Last edit, I promise: For what it's worth, the books appear to get sent correctly.

Last edited by Manichean; 09-22-2010 at 06:08 AM.
Manichean is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Sony collections and custom fields: how to handle duplicates? chaley Calibre 13 09-02-2010 05:11 PM
Amazon integrates Video and Audio with Ebooks luqmaninbmore News 22 06-28-2010 07:22 PM
PRC file doesn't fully import into Calibre MSJim Kindle Formats 1 06-01-2010 03:55 PM
Calibre custom news feed and python help. harrynewman Calibre 4 10-08-2009 10:26 AM
Smashwords integrates with Stanza, optimized for iPhone Smashwords News 1 11-29-2008 04:26 PM


All times are GMT -4. The time now is 10:29 PM.


MobileRead.com is a privately owned, operated and funded community.