[seabream]

As one of the people who requested the inclusion of the new categories for FanFiction.net, thanks for the new functionality. I respect the amount of work that it must have taken, and appreciate the flexibility in how they have been implemented.

Tested on a Macbook Pro running Mac OS X 10.7.4, Calibre 0.8.64 and the beta of FFDL provided in post #844 (updated with notes on 1.6.10), with fics from Fanfiction.net using the following settings in personal.ini:

Spoiler:

is_adult:true

extra_valid_entries:reviews,favs,follows

custom_columns_settings:
reviews=>#reviews,r
favs=>#favs,r
follows=>#follows,r

extra_logpage_entries: reviews,favs,follows
titlepage_entries: series,category,genre,language,characters,status,d atePublished,dateUpdated,dateCreated,rating,warnin gs,numChapters,numWords,site,reviews,favs,follows, description

All of it works as described with multiple fics.

However, Get Story URLs from Web Page seems to now be broken for Fanfiction.net. Attempting results in:
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 251, in get_urls_from_page
NameError: global name 'url' is not defined
Attempted with: http://www.fanfiction.net/u/1728624/ and http://www.fanfiction.net/u/2469838/twitchy-witch
By accident, I tried it with a non-link and got the same result.

I don’t actually read at dramione.org, but since you apparently haven’t gotten many replies on it, I decided to test that too. I made the custom columns listed in custom_columns_settings and used the following basically default settings mentioned in the described personal.ini:

Spoiler:

extra_valid_entries:themes,hermiones,dracos,timeli ne,cliches

include_in_genre:genre, themes
include_in_cliches:hermiones,dracos

themes_label:Themes
hermiones_label:Hermiones
dracos_labelracos
timeline_label:Timeline
cliches_label:Character Cliches

extra_titlepage_entries: themes,timeline,cliches
extra_logpage_entries: themes,timeline,cliches
extra_subject_tags: themes,timeline,cliches

custom_columns_settings:
cliches=>#cliches,r
themes=>#themes,a
timeline=>#timeline,n
hermiones=>#hermiones,a
dracos=>#dracos,a

Picking a random popular author with short stories (ThatRomantic) I tried And What If They Were Lovers ? http://dramione.org/viewstory.php?sid=533 and Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which worked except for Genres. I eventually realized the mistake, but you may want to add to the notes that in order for include_in_genre:genre,themes to work you need to add genre=>#genre under custom_column_settings too, because from the description, at first glance, it looks as though one doesn’t need to do anything in addition. For reference:
## For another example, you could, by uncommenting this line, include
## themes in with genre metadata.
#include_in_genre:genre, themes

Testing with a few more stories by the same author: And What If Karma Caught Up With Them? http://dramione.org/viewstory.php?sid=535 and Battlefield: http://dramione.org/viewstory.php?sid=1479 and I got this from the Updates completed, final status window:
Comments: sequence item 0: expected string or Unicode, NoneType found

If I commented #include_in_genre:genre, themes the comment went away.
Commented or not, Hermiones, Dracos, Timeline and Themes don’t populate with the above two stories. Genre doesn’t give the composite column, but does populate, strangely, with “FanFiction”, which isn’t included with the themes in the Genre column for the story that worked. (I have Genre set to populate with Extra Tags in the GUI Custom Column tab in the FFDL preferences, but when I put include_in_genre:genre,themes,extra_tags and try http://dramione.org/viewstory.php?sid=533 “FanFiction” still doesn’t show, so I’m not sure where it’s coming from)

Not related to new features, but noticed that for Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which of the stories I checked is the only story with an image, even though as shown below I have make_firstimage_cover:false, both under [epub] and [defaults], the epub uses the banner as the title page anyway.

Spoiler:

[epub]
include_images:true
keep_summary_html:true
make_firstimage_cover:false

## This is an example of what your personal configuration might look
## like.

[defaults]
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true

## include images from img tags in the body and summary of
## stories. Images will be converted to jpg for size if possible.
include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
keep_summary_html:true

## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
make_firstimage_cover:false

While I was writing this up, 1.6.10 was released, so I retested on dramione and FanFiction.net. The parts that didn’t work still didn’t. I didn’t check all of the parts that did, but I’ll let you know if I notice anything new.

Also not new to this version, but something I hadn’t tried before. If I call for the FFDL version in the log or title page, with 1.6.10 it shows version 4.4. Same with 1.6.09. Is this intentional?

I can confirm that the relationships meta information is working for me for AO3.

I can’t find the setting for adding a keyboard shortcut for FFDL configuration. Calibre > Preferences > Keyboard > FanFictionDownLoader only has the following options
[Add New FanFiction Book(s) from URL(s), FanFictionDownloader, Get Story URLs from Web Page, Get URLs from Selected Books, Update Existing FanFiction Book(s)] and it doesn’t seem to be anywhere in the FFDL preferences. Is it somewhere else?

Also, I’m not sure exactly when this is was but at least from 1.6.3, fics from FanFiction.net are downloading with left justified text (as displayed on the website) showing up as justified in the epub. I can change it with Convert Books, but that makes everything left justified.

Question about tag substitution. Though I haven’t actually implemented it, in principle I know how to automate tag substitution on import in personal.ini globally, or on a site by site basis with replace_metadata and regexp. But is there a way to have FFDL take a given tag and use different rules for replacement depending on the source category and/or destination column? e.g.: input type: Characters, entry “Han S.” output to Tags as “char Star Wars: Han Solo” and to Characters as “Star Wars.Han Solo”? Designating which metadata category a rule applies to is mostly needed for fandoms where the fandom name is also that of a main character, or a main character’s name tends to show up in story titles, but also because I don’t want to automatically change author names when in fan works, they can contain strings that may also appear in just about any other category. I can see how to do something sort of like it after the fact using Bulk Metadata Edit, though it’s two steps, but I’d need to redo it as I add to the library, and as the library gets larger, running it would take progressively longer, and I’d need to do it separately for every single tag I want to replace (which total in the four digits by now).

Reason being that I use the custom column with heirarchical tags in Calibre, but not all my mobile readers support tags (in three cases, not at all, and in a limited way with the Sony i.e.: it gets very slow above a few hundred tags, and it’s not always accurate, FB Reader on the Samsung is good, but it’s not my phone), which means that I’m using the output of Calibre’s Generate Catalog, which doesn’t include tags from custom columns.

By the way, generally feel free to P.M. me if you have things you want tested. I don’t check the forum every day, or week, and I won’t guarantee that I’ll have time to test immediately, but I’m happy to help when there’s time.