Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 08-26-2013, 12:30 PM   #1681
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by FaceDeer View Post
I just thought of a potential everybody-wins solution: add an "ffdl_last_modified" optional date metadata that users can configure FFDL to populate. That way you can have it update only when FFDL actually modifies the story, in a way that you can configure however you think is optimal for fanfic downloading (for example, you might want to have it only change the date if the *text* of the story changes, ignoring any other metadata updates).
FFDL already has three dates (for most sites) in it's metadata:
  • Published - when the story was first created (from the site)
  • Updated - when the story was last updated (from the site, usually only when new chapters)
  • Created - essentially 'now', when FFDL is being run (labeled Packaged on the title page)

That's what I consider optimal.
JimmXinu is offline   Reply With Quote
Old 08-26-2013, 01:34 PM   #1682
FaceDeer
Connoisseur
FaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enough
 
Posts: 68
Karma: 706
Join Date: Nov 2012
Device: Kobo Touch
Oh, "Created" is changed whenever the epub is modified? I'd thought that field meant "when this story was first added to Calibre" and didn't change after that. That's a piece of information that I didn't need to know for anything and so I ignored it. If it updates whenever the epub file is changed then I'll switch over to using that, it's exactly what I need.
FaceDeer is offline   Reply With Quote
 
Enthusiast
Old 08-26-2013, 01:38 PM   #1683
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by FaceDeer View Post
Oh, "Created" is changed whenever the epub is modified? I'd thought that field meant "when this story was first added to Calibre" and didn't change after that. That's a piece of information that I didn't need to know for anything and so I ignored it. If it updates whenever the epub file is changed then I'll switch over to using that, it's exactly what I need.
Correct. I have no idea now why I labeled it 'Packaged' in place and 'Created' in the other.

It will also be 'now' if you run 'update metadata only'.
JimmXinu is offline   Reply With Quote
Old 08-26-2013, 08:34 PM   #1684
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by JimmXinu View Post
...
So I'll have to change how FFDL does author links. Hopefully I can get a new version out later today, but no promises.
Or not. The methods available at the moment to look up the existing author link are rather onerous (unless I missed one--a distinct possibility).

I've posted additional comments and a couple patches to the bug report. I hope to convince Kovid to include one patch or the other.
JimmXinu is offline   Reply With Quote
Old 08-26-2013, 11:29 PM   #1685
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by JimmXinu View Post
... (unless I missed one--a distinct possibility).

I've posted additional comments and a couple patches to the bug report. I hope to convince Kovid to include one patch or the other.
Now a certainty. Kovid pointed out the correct new_api calls for me to use.

So attached is a version that will only update author links (and therefore last_modified on all those authors' books) when the link actually changes.

Kovid also says "Note that the next release will also not mark books as dirty if you set the same value as before for the author link." But we don't necessarily want to wait that long.

UPDATE Aug 28, 2013 - Remove obsolete beta versions

Last edited by JimmXinu; 08-28-2013 at 11:39 AM. Reason: Remove obsolete beta versions
JimmXinu is offline   Reply With Quote
Old 08-27-2013, 12:58 AM   #1686
Sefiriot
Bujavid tekikin
Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.
 
Sefiriot's Avatar
 
Posts: 137
Karma: 40366
Join Date: Aug 2013
Location: Malaysia
Device: Samsung Galaxy Note N7000
Oooh thank you for the quick work! *makes grabby hands* Will having books marked as dirty do anything nasty to the database though?
Sefiriot is offline   Reply With Quote
Old 08-27-2013, 10:46 AM   #1687
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by Sefiriot View Post
Oooh thank you for the quick work! *makes grabby hands* Will having books marked as dirty do anything nasty to the database though?
No. 'Dirty' in programming terms means something that has been changed and needs to be saved. He just means that the books won't be saved (which is what updates last_modified) if the value is 'set', but isn't actually changed.
JimmXinu is offline   Reply With Quote
Old 08-27-2013, 12:17 PM   #1688
Sefiriot
Bujavid tekikin
Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.
 
Sefiriot's Avatar
 
Posts: 137
Karma: 40366
Join Date: Aug 2013
Location: Malaysia
Device: Samsung Galaxy Note N7000
Quote:
Originally Posted by JimmXinu View Post
No. 'Dirty' in programming terms means something that has been changed and needs to be saved. He just means that the books won't be saved (which is what updates last_modified) if the value is 'set', but isn't actually changed.
Ah, good to know. 'Dirty' can mean different things in various contexts, so I wasn't completely sure what was meant.
Sefiriot is offline   Reply With Quote
Old 08-27-2013, 06:15 PM   #1689
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Version 1.7.38 - 24 Aug 2013
  • Work around for last_modified vs author link update issue in calibre 1.0.0.
  • Better error reporting of metadata update problems.
JimmXinu is offline   Reply With Quote
Old 08-28-2013, 12:20 AM   #1690
FaceDeer
Connoisseur
FaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enough
 
Posts: 68
Karma: 706
Join Date: Nov 2012
Device: Kobo Touch
That better error reporting thing might just have raised its first flags. Testing out my new system using the "Created"/"Packaged" date, I went and grabbed 1200 fanfics off of fanfiction.net that were a mixture of ones I had already and new ones. I ran into 12 instances of the error:

Quote:
calibre, version 1.0.0
ERROR: Error Updating Metadata: <p>An error has occurred while FFDL was updating calibre's metadata for <a href='http://www.fanfiction.net/s/9587231/1/'>The Legend of the Titans</a>.</p>The ebook has been updated, but the metadata has not.

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 1117, in update_books_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 1535, in update_metadata
KeyError: None
However, inspecting several of the fanfics with errors afterward seemed to show that the metadata in my database had indeed been updated to match what was on the site. So maybe these are false positives?

I tried redownloading the twelve and the errors all popped up again, so this is repeatable for me at least. The fanfics in question are:

Edit: Unfortunately the new error reporting hasn't turned up anything regarding my trouble with FFDL on my laptop, it still just quietly hangs at the "updating calibre with fanfiction stories" step. I figure I'll wait until weekend to try out the next Calibre update, to see if anything gets fixed in that that affects this, and then try the tear-it-all-down-and-rebuild approach.

Last edited by FaceDeer; 08-28-2013 at 01:59 AM. Reason: nothing new on my laptop
FaceDeer is offline   Reply With Quote
Old 08-28-2013, 10:39 AM   #1691
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by FaceDeer View Post
That better error reporting thing might just have raised its first flags.

I tried redownloading the twelve and the errors all popped up again, so this is repeatable for me at least. The fanfics in question are:
All of those stories have an author with '&' or 'and' in their author name. Calibre is automatically splitting that into two different authors. So when FFDL tries to look up the author to set their link, it fails.

It looks like it's always done that and before the attempt to set the author link just failed silently.

Attached is a temporary version that gets it past error, but doesn't fix the fundamental problem.

I'll need to research that a bit more first--I'm not sure how to get calibre to accept '&' or 'and' in a single author's name.

UPDATE Aug 28, 2013 - Remove obsolete beta versions

Last edited by JimmXinu; 08-28-2013 at 11:39 AM. Reason: Remove obsolete beta versions
JimmXinu is offline   Reply With Quote
Old 08-28-2013, 11:13 AM   #1692
FaceDeer
Connoisseur
FaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enough
 
Posts: 68
Karma: 706
Join Date: Nov 2012
Device: Kobo Touch
http://www.fanfiction.net/s/9622698/1/ has an author name with "with" in it instead of "and" or "&", Calibre might be splitting on that too.

What a clever program.
FaceDeer is offline   Reply With Quote
Old 08-28-2013, 11:26 AM   #1693
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,223
Karma: 5495470
Join Date: Aug 2009
Location: The (original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by FaceDeer View Post
http://www.fanfiction.net/s/9622698/1/ has an author name with "with" in it instead of "and" or "&", Calibre might be splitting on that too.

What a clever program.
There is a tweak for this:

Code:
ID: authors_split_regex

By default, calibre splits a string containing multiple author names on
ampersands and the words "and" and "with". You can customize the splitting
by changing the regular expression below. Strings are split on whatever the
specified regular expression matches, in addition to ampersands.
Default: r'(?i),?\s+(and|with)\s+'
theducks is offline   Reply With Quote
Old 08-28-2013, 11:37 AM   #1694
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by theducks View Post
There is a tweak for this:

Code:
ID: authors_split_regex

By default, calibre splits a string containing multiple author names on
ampersands and the words "and" and "with". You can customize the splitting
by changing the regular expression below. Strings are split on whatever the
specified regular expression matches, in addition to ampersands.
Default: r'(?i),?\s+(and|with)\s+'
Yep, I saw that.

But that doesn't allow authors with '&' (that part is hard coded) and tweaks affect all libraries and books.

(FYI, calibre changes '&' in an author name to ';'. There's no working around that.)
JimmXinu is offline   Reply With Quote
Old 08-28-2013, 11:38 AM   #1695
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,584
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Here is a version that I think addresses the author splitting issue.

UPDATE Aug 31 2013 -- Remove obsolete beta versions

Last edited by JimmXinu; 08-31-2013 at 04:37 PM. Reason: Remove obsolete beta versions
JimmXinu is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Count Pages kiwidude Plugins 640 07-14-2014 04:06 AM
[GUI Plugin] Resize Cover kiwidude Plugins 34 07-13-2014 07:15 PM
[GUI Plugin] Find Duplicates kiwidude Plugins 472 07-11-2014 09:34 AM
[GUI Plugin] Open With kiwidude Plugins 223 07-09-2014 02:42 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 09:07 AM.


MobileRead.com is a privately owned, operated and funded community.