Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 09-20-2012, 10:21 PM   #856
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Hmm. Well, what I was thinking about wouldn't have caught that anyway--that's a regexp issue, not just an ini file problem. I've added that notation to the wishlist item, though.
JimmXinu is offline   Reply With Quote
Old 09-21-2012, 02:00 PM   #857
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Version 1.6.10 - 21 Sep 2012

New sites (thanks Marillapm):
  • bloodties-fan.com
  • thehookupzone.net

This version contains some metadata features that have been requested, plus quite a bit of internal refactoring to allow them.

New Features:
  • Allow individual site adapters to add extra metadata entries to the standard ones. (key word: extra_valid_entries)
  • Allow custom labels on title/log pages for extra metadata entries. (key words: themes_label, cliches_label, etc)
  • Allow those extra metadata to be included in titlepage/logpage, or added to existing metadata entries. (key words: extra_titlepage_entries, extra_logpage_entries, include_in_genre, etc)
  • Allow new, purely composite, metadata entries to be made from existing and extra metadata entries. (key words: include_in_*)
  • Allow existing metadata entries to be redefined. If one site has what you think should be category and genre reversed, for example, you could swap them. (key words: include_in_category, include_in_genre, etc)
  • Allow calibre custom columns to be populated from the new extra metadata entries. (key word: custom_columns_settings)
  • New configuration options to add to titlepage_entries, logpage_entries or include_subject_tags for specific sites (instead of replace it). (key words: extra_titlepage_entries, extra_logpage_entries, extra_subject_tags)
  • Allow the format for Published, Updated and Packaged dates on titlepage/logpage to be configured in personal.ini. (key words: dateCreated_format, datePublished_format, dateUpdated_format)
  • Cover support for dramione.org.
  • A fix for squidge.org/peja Rating.
  • Loosen url check on AO3 to allow more copy/paste URLs.
  • Fix author on castlefans.org due to site/skin changes.

For now, I've only added a few site specific metadata that I know have been asked for. I repeat: only a few site specific metadata listed below have been added so far.

New 'Extra' Metadata:
  • dramione.org: themes, hermiones, dracos, timeline, cliches (cliches = hermiones, dracos to demonstrate the composite feature)
  • www.fanfiction.net: reviews, favs, follows
  • www.fictionpress.com: reviews, favs, follows (fanfiction.net & fictionpress.com share code, but not configuration)

dramione.org is the prototype for this feature; ini code:
Spoiler:
Code:
[dramione.org]

## Some adapters collect additional meta information beyond the
## standard ones.  They need to be defined in extra_valid_entries to
## tell the rest of the FFDL system about them.  They can be used in
## include_subject_tags, titlepage_entries, extra_titlepage_entries,
## logpage_entries, extra_logpage_entries, and include_in_* config
## items.  You can also add additional entries here to build up
## composite metadata entries.  dramione.org, for example, adds
## 'cliches' and then defines as the composite of hermiones,dracos in
## include_in_cliches.
extra_valid_entries:themes,hermiones,dracos,timeline,cliches
include_in_cliches:hermiones,dracos

## For another example, you could, by uncommenting this line, include
## themes in with genre metadata.
#include_in_genre:genre, themes

## You can give each new valid entry a specific label for use on
## titlepage and logpage.  If not defined, it will simply be the 
themes_label:Themes
hermiones_label:Hermiones
dracos_label:Dracos
timeline_label:Timeline
cliches_label:Character Cliches

## extra_titlepage_entries (and extra_logpage_entries) *add* to
## titlepage_entries (and logpage_entries) so you can add site
## specific entries to titlepage/logpage without having to copy the
## entire titlepage_entries line.  (But if you want them higher than
## the end, you will need to copy titlepage_entries.)
extra_titlepage_entries: themes,timeline,cliches
extra_logpage_entries: themes,timeline,cliches
extra_subject_tags: themes,timeline,cliches

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.)  There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'.  'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only.
## Make sure to keep at least one space at the start of each
## line.

custom_columns_settings:
 cliches=>#acolumn,r
 themes=>#bcolumn,a
 timeline=>#ccolumn,n


Assuming this works well, I'll take requests on extra metadata to add for other sites. I'm not, however, planning to add new metadata across the board. It's easy to say 'add reviews for all sites', but to actually add it to all 75 adapters is not.

Jim
JimmXinu is offline   Reply With Quote
Old 09-21-2012, 04:36 PM   #858
crackshot91
Enthusiast
crackshot91 has learned how to buy an e-book online
 
Posts: 40
Karma: 98
Join Date: Jan 2011
Device: Kindle 3G
Hey, Jimm. I'm getting a strange error, here. I've downloaded Hundreds of stories over the past week, but for these 4 stories (And only these 4), I get the error "'NoneType' object has no attribute 'group'"

Here are the story urls:

http://www.fimfiction.net/story/6270/Cheater
http://www.fimfiction.net/story/6012/Had-a-Rough-Day
http://www.fimfiction.net/story/5827...-What-They-Say
http://www.fimfiction.net/story/5824/Twilight-Mornings

I thought at first that it possibly had something to do with the author's name? Perhaps the period character is the cause? But, I am able to download this story:

http://www.fimfiction.net/story/15325/Something-Secret

Which is by the same author as the first 4.

This is what I see when I try to download Mr.V's stories:



I will say that about a month ago, I was able to download Mr.V's stories without problems, so I'm wondering if perhaps a recent change to the site, or perhaps to the plugin, jostled something in the code. :P

If you could look into the problem, I'd be grateful.

(BTW, the changes you made for the bbcode/html description issue and the fimfiction covers issue have worked wonderfully. I've had no issues since.)
crackshot91 is offline   Reply With Quote
Old 09-21-2012, 05:17 PM   #859
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by crackshot91 View Post
Hey, Jimm. I'm getting a strange error, here. I've downloaded Hundreds of stories over the past week, but for these 4 stories (And only these 4), I get the error "'NoneType' object has no attribute 'group'"

...

(BTW, the changes you made for the bbcode/html description issue and the fimfiction covers issue have worked wonderfully. I've had no issues since.)
Au contraire! The bbcode decoding is exactly what's causing your new problem. That particular author has put their own 'tags' in ("[ 2nd person ]" etc), which is causing the bbcode decoder to fail.

I've made a tweak to the bbcode parser that I think fixes it. Please give this version a try.

Last edited by JimmXinu; 10-02-2012 at 10:49 AM. Reason: Remove obsolete beta versions
JimmXinu is offline   Reply With Quote
Old 09-22-2012, 01:17 AM   #860
crackshot91
Enthusiast
crackshot91 has learned how to buy an e-book online
 
Posts: 40
Karma: 98
Join Date: Jan 2011
Device: Kindle 3G
Quote:
Originally Posted by JimmXinu View Post
Au contraire! The bbcode decoding is exactly what's causing your new problem. That particular author has put their own 'tags' in ("[ 2nd person ]" etc), which is causing the bbcode decoder to fail.

I've made a tweak to the bbcode parser that I think fixes it. Please give this version a try.


You rock.

EDIT: Yep, it's working fine now. So it was a plugin update that messed things up!

(I am so sorry for that I appreciate what you do oh god please don't be mad)

Last edited by crackshot91; 09-22-2012 at 01:20 AM.
crackshot91 is offline   Reply With Quote
Old 09-22-2012, 11:03 AM   #861
seabream
Enthusiast
seabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheese
 
Posts: 48
Karma: 1056
Join Date: Jul 2012
Location: Central Canada
Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX
1.6.10 feature test

As one of the people who requested the inclusion of the new categories for FanFiction.net, thanks for the new functionality. I respect the amount of work that it must have taken, and appreciate the flexibility in how they have been implemented.

Tested on a Macbook Pro running Mac OS X 10.7.4, Calibre 0.8.64 and the beta of FFDL provided in post #844 (updated with notes on 1.6.10), with fics from Fanfiction.net using the following settings in personal.ini:

Spoiler:
is_adult:true

extra_valid_entries:reviews,favs,follows

custom_columns_settings:
reviews=>#reviews,r
favs=>#favs,r
follows=>#follows,r

extra_logpage_entries: reviews,favs,follows
titlepage_entries: series,category,genre,language,characters,status,d atePublished,dateUpdated,dateCreated,rating,warnin gs,numChapters,numWords,site,reviews,favs,follows, description


All of it works as described with multiple fics.

However, Get Story URLs from Web Page seems to now be broken for Fanfiction.net. Attempting results in:
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 251, in get_urls_from_page
NameError: global name 'url' is not defined
Attempted with: http://www.fanfiction.net/u/1728624/ and http://www.fanfiction.net/u/2469838/twitchy-witch
By accident, I tried it with a non-link and got the same result.

I don’t actually read at dramione.org, but since you apparently haven’t gotten many replies on it, I decided to test that too. I made the custom columns listed in custom_columns_settings and used the following basically default settings mentioned in the described personal.ini:
Spoiler:

extra_valid_entries:themes,hermiones,dracos,timeli ne,cliches

include_in_genre:genre, themes
include_in_cliches:hermiones,dracos

themes_label:Themes
hermiones_label:Hermiones
dracos_labelracos
timeline_label:Timeline
cliches_label:Character Cliches

extra_titlepage_entries: themes,timeline,cliches
extra_logpage_entries: themes,timeline,cliches
extra_subject_tags: themes,timeline,cliches

custom_columns_settings:
cliches=>#cliches,r
themes=>#themes,a
timeline=>#timeline,n
hermiones=>#hermiones,a
dracos=>#dracos,a

Picking a random popular author with short stories (ThatRomantic) I tried And What If They Were Lovers ? http://dramione.org/viewstory.php?sid=533 and Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which worked except for Genres. I eventually realized the mistake, but you may want to add to the notes that in order for include_in_genre:genre,themes to work you need to add genre=>#genre under custom_column_settings too, because from the description, at first glance, it looks as though one doesn’t need to do anything in addition. For reference:
## For another example, you could, by uncommenting this line, include
## themes in with genre metadata.
#include_in_genre:genre, themes

Testing with a few more stories by the same author: And What If Karma Caught Up With Them? http://dramione.org/viewstory.php?sid=535 and Battlefield: http://dramione.org/viewstory.php?sid=1479 and I got this from the Updates completed, final status window:
Comments: sequence item 0: expected string or Unicode, NoneType found

If I commented #include_in_genre:genre, themes the comment went away.
Commented or not, Hermiones, Dracos, Timeline and Themes don’t populate with the above two stories. Genre doesn’t give the composite column, but does populate, strangely, with “FanFiction”, which isn’t included with the themes in the Genre column for the story that worked. (I have Genre set to populate with Extra Tags in the GUI Custom Column tab in the FFDL preferences, but when I put include_in_genre:genre,themes,extra_tags and try http://dramione.org/viewstory.php?sid=533 “FanFiction” still doesn’t show, so I’m not sure where it’s coming from)

Not related to new features, but noticed that for Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which of the stories I checked is the only story with an image, even though as shown below I have make_firstimage_cover:false, both under [epub] and [defaults], the epub uses the banner as the title page anyway.
Spoiler:

[epub]
include_images:true
keep_summary_html:true
make_firstimage_cover:false

## This is an example of what your personal configuration might look
## like.

[defaults]
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true

## include images from img tags in the body and summary of
## stories. Images will be converted to jpg for size if possible.
include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
keep_summary_html:true

## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
make_firstimage_cover:false

While I was writing this up, 1.6.10 was released, so I retested on dramione and FanFiction.net. The parts that didn’t work still didn’t. I didn’t check all of the parts that did, but I’ll let you know if I notice anything new.

Also not new to this version, but something I hadn’t tried before. If I call for the FFDL version in the log or title page, with 1.6.10 it shows version 4.4. Same with 1.6.09. Is this intentional?

I can confirm that the relationships meta information is working for me for AO3.

I can’t find the setting for adding a keyboard shortcut for FFDL configuration. Calibre > Preferences > Keyboard > FanFictionDownLoader only has the following options
[Add New FanFiction Book(s) from URL(s), FanFictionDownloader, Get Story URLs from Web Page, Get URLs from Selected Books, Update Existing FanFiction Book(s)] and it doesn’t seem to be anywhere in the FFDL preferences. Is it somewhere else?

Also, I’m not sure exactly when this is was but at least from 1.6.3, fics from FanFiction.net are downloading with left justified text (as displayed on the website) showing up as justified in the epub. I can change it with Convert Books, but that makes everything left justified.

Question about tag substitution. Though I haven’t actually implemented it, in principle I know how to automate tag substitution on import in personal.ini globally, or on a site by site basis with replace_metadata and regexp. But is there a way to have FFDL take a given tag and use different rules for replacement depending on the source category and/or destination column? e.g.: input type: Characters, entry “Han S.” output to Tags as “char Star Wars: Han Solo” and to Characters as “Star Wars.Han Solo”? Designating which metadata category a rule applies to is mostly needed for fandoms where the fandom name is also that of a main character, or a main character’s name tends to show up in story titles, but also because I don’t want to automatically change author names when in fan works, they can contain strings that may also appear in just about any other category. I can see how to do something sort of like it after the fact using Bulk Metadata Edit, though it’s two steps, but I’d need to redo it as I add to the library, and as the library gets larger, running it would take progressively longer, and I’d need to do it separately for every single tag I want to replace (which total in the four digits by now).

Reason being that I use the custom column with heirarchical tags in Calibre, but not all my mobile readers support tags (in three cases, not at all, and in a limited way with the Sony i.e.: it gets very slow above a few hundred tags, and it’s not always accurate, FB Reader on the Samsung is good, but it’s not my phone), which means that I’m using the output of Calibre’s Generate Catalog, which doesn’t include tags from custom columns.

By the way, generally feel free to P.M. me if you have things you want tested. I don’t check the forum every day, or week, and I won’t guarantee that I’ll have time to test immediately, but I’m happy to help when there’s time.

Last edited by seabream; 09-22-2012 at 11:13 AM. Reason: typos and adding spoiler tags for length
seabream is offline   Reply With Quote
Old 09-22-2012, 12:38 PM   #862
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by crackshot91 View Post


You rock.

EDIT: Yep, it's working fine now. So it was a plugin update that messed things up!

(I am so sorry for that I appreciate what you do oh god please don't be mad)
Naw, I was just being over dramatic for humorous effect and forgot the smiley.
JimmXinu is offline   Reply With Quote
Old 09-22-2012, 02:21 PM   #863
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by seabream View Post
As one of the people who requested the inclusion of the new categories for FanFiction.net, thanks for the new functionality. I respect the amount of work that it must have taken, and appreciate the flexibility in how they have been implemented.
Many thanks for taking the time to do this detailed testing. It really helps me out a lot.

Quote:
Originally Posted by seabream View Post
Tested on a Macbook Pro running Mac OS X 10.7.4, Calibre 0.8.64 and the beta of FFDL provided in post #844 (updated with notes on 1.6.10), with fics from Fanfiction.net ...

All of it works as described with multiple fics.
Excellent.

Quote:
Originally Posted by seabream View Post
However, Get Story URLs from Web Page seems to now be broken...
So it is. That's my bad--I don't use that feature so I forgot to test it.

Quote:
Originally Posted by seabream View Post
I don’t actually read at dramione.org, but since you apparently haven’t gotten many replies on it, I decided to test that too.
I appreciate that greatly, thank you.

Quote:
Originally Posted by seabream View Post
Picking a random popular author with short stories (ThatRomantic) I tried And What If They Were Lovers ? http://dramione.org/viewstory.php?sid=533 and Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which worked except for Genres. I eventually realized the mistake, but you may want to add to the notes that in order for include_in_genre:genre,themes to work you need to add genre=>#genre under custom_column_settings too, because from the description, at first glance, it looks as though one doesn’t need to do anything in addition. For reference:
## For another example, you could, by uncommenting this line, include
## themes in with genre metadata.
#include_in_genre:genre, themes
This part appears to be working as designed.

'include_in_genre' adds the theme list to the genre list. The idea is that if you already have #genre column defined and have set FFDL to populate from 'genre' on the Custom Columns tab (not custom_column_settings), that will add themes to it, too.

If you uncomment include_in_genre:genre, themes, both genre and themes appear after the Genre heading on the title page and in custom columns set to populate from 'genre' on the Custom Columns tab.

I'm happy to change the documenting comment to be more clear, do you have any suggestions?

Quote:
Originally Posted by seabream View Post
Testing with a few more stories by the same author: And What If Karma Caught Up With Them? http://dramione.org/viewstory.php?sid=535 and Battlefield: http://dramione.org/viewstory.php?sid=1479 and I got this from the Updates completed, final status window:
Comments: sequence item 0: expected string or Unicode, NoneType found
I only see that problem with Battlefield: http://dramione.org/viewstory.php?sid=1479, but that's enough to find it--it's a bug that only happens when using custom_columns_settings to set a column with 'a' flag and the story doesn't have any of that--Battlefield doesn't have any themes.

Quote:
Originally Posted by seabream View Post
If I commented #include_in_genre:genre, themes the comment went away.
Commented or not, Hermiones, Dracos, Timeline and Themes don’t populate with the above two stories. Genre doesn’t give the composite column, but does populate, strangely, with “FanFiction”, which isn’t included with the themes in the Genre column for the story that worked. (I have Genre set to populate with Extra Tags in the GUI Custom Column tab in the FFDL preferences, but when I put include_in_genre:genre,themes,extra_tags and try http://dramione.org/viewstory.php?sid=533 “FanFiction” still doesn’t show, so I’m not sure where it’s coming from)
I'm not able to replicate this. It might have been a side effect from the 'a' flag bug above. As for how "FanFiction" could end up in genre, I suspect it was an issue with it being set from Custom Column tab (which happens before custom_columns_settings are applied), and then added to? And it's 'extratags' not 'extra_tags', to use in include_in_genre.

Quote:
Originally Posted by seabream View Post
Not related to new features, but noticed that for Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which of the stories I checked is the only story with an image, even though as shown below I have make_firstimage_cover:false, both under [epub] and [defaults], the epub uses the banner as the title page anyway.
The banner is being treated as an explicit cover image, which is different than make_firstimage_cover. There's a very few sites (fimfic, ffnet and dramione) that have explicit cover images. The only ways to prevent it are never_make_cover: true or include_images: false.

I'm open to suggestions if this doesn't seem like the correct behavior.

Quote:
Originally Posted by seabream View Post
Also not new to this version, but something I hadn’t tried before. If I call for the FFDL version in the log or title page, with 1.6.10 it shows version 4.4. Same with 1.6.09. Is this intentional?
Sort of. FFDL started out as a separate project and still has CLI and web service versions that run off the same code base. When I made the plugin, I started it at 1.0.0 instead of matching the 4.4 version of the existing project. I'll think about how I might get that to show the plugin version instead. Thanks for pointing it out.

Quote:
Originally Posted by seabream View Post
I can confirm that the relationships meta information is working for me for AO3.
Excellent.

Quote:
Originally Posted by seabream View Post
I can’t find the setting for adding a keyboard shortcut for FFDL configuration. Calibre > Preferences > Keyboard > FanFictionDownLoader only has the following options
[Add New FanFiction Book(s) from URL(s), FanFictionDownloader, Get Story URLs from Web Page, Get URLs from Selected Books, Update Existing FanFiction Book(s)] and it doesn’t seem to be anywhere in the FFDL preferences. Is it somewhere else?
Good point. I'd forgotten that the code I use to the create the menu (that caused some Macs crash, I'm sure you remember) is also what registers shortcuts with calibre. I'll try to get the shortcuts for you, but there's a chance it will bump into the Mac crash issue again.

Quote:
Originally Posted by seabream View Post
Also, I’m not sure exactly when this is was but at least from 1.6.3, fics from FanFiction.net are downloading with left justified text (as displayed on the website) showing up as justified in the epub. I can change it with Convert Books, but that makes everything left justified.
Check out the output_css ini parameter. Quite frankly, the default is text-align: justify just because I like that better.

Spoiler:
Code:
[epub]
## output background color--only used by html and epub (and ignored in
## epub by many readers).  Included below in output_css--will be
## ignored if not in output_css.
background_color: ffffff

## Allow customization of CSS.  Make sure to keep at least one space
## at the start of each line and to escape % to %%.  Also need
## background_color to be in the same section, if included in CSS.
## 'adobe-text-layout: optimizeSpeed;' prevents hyphenation on newer Nooks
output_css: 
 body { background-color: #%(background_color)s;
        text-align: justify;
        margin: 2%%;
	adobe-text-layout: optimizeSpeed; }
 pre { font-size: x-small; }
 sml { font-size: small; }
 h1 { text-align: center; }
 h2 { text-align: center; }
 h3 { text-align: center; }
 h4 { text-align: center; }
 h5 { text-align: center; }
 h6 { text-align: center; }
 .CI {
     text-align:center;
     margin-top:0px;
     margin-bottom:0px;
     padding:0px;
     }
 .center   {text-align: center;}
 .cover    {text-align: center;}
 .full     {width: 100%%; }
 .quarter  {width: 25%%; }
 .smcap    {font-variant: small-caps;}
 .u        {text-decoration: underline;}
 .bold     {font-weight: bold;}


Quote:
Originally Posted by seabream View Post
Question about tag substitution. Though I haven’t actually implemented it, in principle I know how to automate tag substitution on import in personal.ini globally, or on a site by site basis with replace_metadata and regexp. But is there a way to have FFDL take a given tag and use different rules for replacement depending on the source category and/or destination column? e.g.: input type: Characters, entry “Han S.” output to Tags as “char Star Wars: Han Solo” and to Characters as “Star Wars.Han Solo”?
Not at this time. I admit it's a bit of a ham-fisted approach at the moment. However, with the refactoring that happened for the stuff above, I think it might be doable now for source metadata (characters, genre, etc). Changes based on destination column would be a whole different thing, but I think about it.

Quote:
Originally Posted by seabream View Post
By the way, generally feel free to P.M. me if you have things you want tested. I don’t check the forum every day, or week, and I won’t guarantee that I’ll have time to test immediately, but I’m happy to help when there’s time.
I appreciate that, I'll keep it in mind.

Many thanks for your help. Hopefully I can get out a version addressing these issues today yet.
JimmXinu is offline   Reply With Quote
Old 09-22-2012, 04:07 PM   #864
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
This version fixes the 'Get Story URLs from Web Page' issue, and the problem when using custom_columns_settings to set a column with 'a' flag and the story doesn't have any of that particular metadata.

This also introduces a different way to deal with the problem of Mac users crashing calibre.

To recap: A couple of Mac users have experienced calibre crashes after installing FFDL PI. Long story short, I removed the Configure Plugin and About Plugin menu items for all Mac users to fix it for one user.

It has since occurred to me that I have no idea how many Mac users FFDL PI actually has and how many of them have this problem. Am I inconveniencing a bunch of people for a problem only one user has?

So I came up with a different way remove those menu items for those users it crashes for. Create a file named 'fanfictiondownloader_macmenuhack.txt' in your calibre/plugins directory. If some Mac user could post an example of where that typically is on Mac, I'd appreciate it.

(Theoretically, it should be possible to figure out why it only happens to FFDL and fix/workaround it. But without having a Mac that's doing it to experiment on--or at least a user with one willing to try dozens of versions--I don't see that happening.)

Please let me know how this version works for you.

Thanks,
Jim

Last edited by JimmXinu; 10-02-2012 at 06:25 PM. Reason: Remove obsolete beta versions
JimmXinu is offline   Reply With Quote
Old 09-23-2012, 05:56 PM   #865
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
No feedback on the previous test version yet, hopefully that means it's working fine.

This version improves the handling of include_in_* config so that include_in_* lines can now include each other.

It also adds the ability to restrict replace_metadata lines to only modify specific metadata by entry/key word.

Code:
replace_metadata:
 genre=>Sci-Fi=>SF
 genre=>Comedy=>Humor
 characters,ships=>Harry P\.=>Harry Potter
(FYI, the '.' in 'Harry P.' needs to be escaped with '\' because '.' means 'any character' in regular expression.)

By using extra_valid_entries, I believe you should be able to do anything you would be able to if I had implemented replacement by destination column.

Code:
[somesite]

extra_valid_entries:editedchars
editedchars_label:Characters by site

include_in_editedchars:characters

replace_metadata:
 editedchars=>^(.+)$=>somesite:\1

extra_titlepage_entries: editedchars

custom_columns_settings:
 editedchars=>#hierchars,r

Last edited by JimmXinu; 10-02-2012 at 06:25 PM. Reason: Remove obsolete beta versions
JimmXinu is offline   Reply With Quote
Old 09-24-2012, 03:26 AM   #866
seabream
Enthusiast
seabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheese
 
Posts: 48
Karma: 1056
Join Date: Jul 2012
Location: Central Canada
Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX
@863, 864, 865

Sorry, I’m not generally going to have as quick a response time as you do. It’s nice, but it’s not something I can manage all the time myself.
Quote:
Originally Posted by JimmXinu View Post
Many thanks for taking the time to do this detailed testing. It really helps me out a lot.
You’re welcome. Of course it also helps other users and me too, so hey.

Quote:
Originally Posted by JimmXinu View Post
include_in_genre:genre, themes
This part appears to be working as designed.
Right. I realised that was how it was supposed to work after the fact. There’s enough there that it can be worked out. It was really my mistake confusing ‘genre’, the metadata category, ’Genre’, the title page heading, and ‘Genre’ the custom column. While it can be explicitly specified in documentation, things then get unwieldly, so maybe it’s best left as is.

Quote:
Originally Posted by JimmXinu View Post
This version fixes the 'Get Story URLs from Web Page' issue, and the problem when using custom_columns_settings to set a column with 'a' flag and the story doesn't have any of that particular metadata.

This also introduces a different way to deal with the problem of Mac users crashing calibre.
I’ve re-tested Battlefield. The error message, and failure to populate of the columns mentioned are both gone.
Re-tested Get URLs from Web Page, and it works properly too.

The menu hack works, as far as removing the menu items goes. It’s not perfect since it also disables shortcut keys for those items if they’ve been set prior. Either way, it’s not crashing yet, which I do indeed remember, so that’s all good.
The Calibre plugin folder is at ~/Library/Preferences/calibre/plugins where ‘~’ is the user’s home folder, which is generally (though not always) called their username, and located at Primary Hard Drive/Users/
Note that in OS X 10.7 and up, users’ Library folders (as distinct from the system Library folder) are hidden. One can get there in Finder by clicking Go > Go to Folder… in the menu bar and entering ~/Library/Preferences/calibre/plugins in the dialogue box. The default keyboard shortcut for Go to Folder is Command + Shift + G

Quote:
Originally Posted by JimmXinu View Post
The banner is being treated as an explicit cover image, which is different than make_firstimage_cover. There's a very few sites (fimfic, ffnet and dramione) that have explicit cover images. The only ways to prevent it are never_make_cover: true or include_images: false.

I'm open to suggestions if this doesn't seem like the correct behavior.
That makes sense, but does that mean that there’s no way to call for the image to appear in the epub, but after the summary titlepage image (other than manually adding it afterwards)? As you note, one generally doesn’t want the ffnet thumbnail as the cover; but one might want to see it.

Quote:
Originally Posted by JimmXinu View Post
FFDL started out as a separate project and still has CLI and web service versions that run off the same code base. When I made the plugin, I started it at 1.0.0 instead of matching the 4.4 version of the existing project. I'll think about how I might get that to show the plugin version instead. Thanks for pointing it out.
You’re welcome. One of these days I might check out the other versions just to see how they work.

Quote:
Originally Posted by JimmXinu View Post
Check out the output_css ini parameter. Quite frankly, the default is text-align: justify just because I like that better.
Oh. I omitted that section from personal.ini because I wanted to preserve original formatting. Does the default override anyway? Aside from finding left justified easier to read (which I can specify with this, or Convert books), some stories with poetry, lyrics, etc… shouldn’t have the entire body in a given justification, which I don’t know how to retain using that parameter.

Quote:
Originally Posted by JimmXinu View Post
This version improves the handling of include_in_* config so that include_in_* lines can now include each other.

It also adds the ability to restrict replace_metadata lines to only modify specific metadata by entry/key word.
Very cool! Thanks. I’ll try it out and get back to you.

You mentioned taking metadata requests: Could you add support for isolating what AO3 calls “Category” on the story page itself, and “Relationships, pairings, orientations” in the symbols key (F/F, Gen, F/M, Multi, Other, M/M) as its own metadata source? Currently, it’s lumped in with extra tags, but that means that I can end up with a field that says, for example: “Angst, Insanity, truly bad and wrong, F/M,” which is less useful for sorting purposes. There was a brief period where I could call it as “Category” some months ago, but now that pulls Fandom on AO3. I don’t want to a custom column set to “Text, with fixed set of permitted values”, because there are other classes of relationships I want to include from other sites without adding additional columns. If it’s not too greedy, I’d also like to request counts for kudos, comments, bookmarks, and hits on AO3.
seabream is offline   Reply With Quote
Old 09-24-2012, 09:37 AM   #867
CyberCrisis
Junior Member
CyberCrisis began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Jul 2012
Device: none
Jim seem to be having problems with yourfanfiction.com FFDL seem to no longer be able to recognize new urls. It will updated the ones form book already made but when you try to put in a new one it does nothing.
CyberCrisis is offline   Reply With Quote
Old 09-24-2012, 09:53 AM   #868
seabream
Enthusiast
seabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheese
 
Posts: 48
Karma: 1056
Join Date: Jul 2012
Location: Central Canada
Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX
@865, testing restricted replace_metadata

Currently just tested with ffnet
FFDL Version now shows up properly in titlepage and logpage

replace_metadata is beautiful (after some trial and error figuring out how to configure things). Thank you so much. Once I actually put all the various tags in, this will save so much time and reduce data entry errors from manual substitution.

For anyone who might want some examples, here’s an illustrative configuration. Obviously add categories as desired. EDIT: for some reason, when posted, there are random spaces inserted here and there in the spoilered block. They don't appear in the edit window, so I don't know how to get rid of them, so if you use the code, proofread it.
Spoiler:
[www.fanfiction.net]

extra_valid_entries:reviews,favs,follows,editedcha rs,editedfand,
editedchars_label:Characters by fandom
editedfand_label:Fandom

include_in_editedchars:characters
include_in_editedfand:category

custom_columns_settings:
reviews=>#reviews,r
favs=>#favs,r
follows=>#follows,r
characters=>#relationship,n
editedchars=>#characters,n
editedfand=>#fandom,n

replace_metadata:
characters=>Rachel B\.=>Rachel Berry
characters=>Quinn F\.=>Quinn Fabray
characters=>(Rachel Berry|Quinn Fabray)=>ch Glee: \1
#prepend character and fandom designations for alphabetical tag sort in Generate Catalog’s Calibre catalog
editedchars=>(ch Glee: )=>Glee.
##alter prepend for inclusion in heirarchical Characters column
category=>^(.+)$=>fand: \1
##prepend fand: to fandom tags for inclusion in Tags column for alphabetical tag sort
editedfand=>(fand: )=>
##remove prepend for listing in Fandom column, titlepage and logpage

logpage_entries: dateCreated,datePublished,dateUpdated,numChapters, numWords,status,title,author,editedchars,ships,ser ies,description,editedfand,category,rating,warning s,version,reviews,favs,follows
titlepage_entries: series,editedfand,category,language,editedchars,sh ips,status,datePublished,dateUpdated,dateCreated,r ating,warnings,numChapters,numWords,site,reviews,f avs,follows,description,extratags,storyId,authorId ,version


I’ll get back to you on test results with AO3 when I’ve got them, but I don't expect big differences.

Last edited by seabream; 09-24-2012 at 11:59 AM. Reason: added note about strange spacing.
seabream is offline   Reply With Quote
Old 09-24-2012, 11:03 AM   #869
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by CyberCrisis View Post
Jim seem to be having problems with yourfanfiction.com FFDL seem to no longer be able to recognize new urls. It will updated the ones form book already made but when you try to put in a new one it does nothing.
Example URLs?
JimmXinu is offline   Reply With Quote
Old 09-24-2012, 01:11 PM   #870
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
Been on a fanfic reading binge/marathon so didn't realize all these changes are being made to FFDL. Will try to get around to testing when I find time. Thank you very much!
ilovejedd is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Count Pages kiwidude Plugins 668 Today 10:13 AM
[GUI Plugin] Open With kiwidude Plugins 232 10-09-2014 12:38 AM
[GUI Plugin] Find Duplicates kiwidude Plugins 508 09-02-2014 07:00 PM
[GUI Plugin] Resize Cover kiwidude Plugins 34 07-13-2014 07:15 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 09:55 PM.


MobileRead.com is a privately owned, operated and funded community.