[GUI Plugin] FanFictionDownLoader

[Amalthia]

Originally Posted by JimmXinu

replace_br_with_p shouldn't effect font size unless your CSS lists different size for text in p tags vs other tags.

If you have keep_style_attr set to true, there may also be style attributes from the original site coming through.

Without seeing the source story, your personal.ini and the epub generated, all I can do is speculate.

Here is my personal.ini info:

[archiveofourown.org]
username:
password:
is_adult:true
extra_titlepage_entries: collections
replace_br_with_p:true
extra_valid_entries:fandoms,comments,kudos,collect ions
fandoms_label:Fandoms
comments_label:Comments
collections_label:Collections

custom_columns_settings:
comments=>#reviewcount,r
collections=>#fandomchallenges,r

## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.) There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'. 'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only. (Default is 'r'.)
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

#custom_columns_settings:
# cliches=>#acolumn,r
# themes=>#bcolumn,a
# timeline=>#ccolumn,n



## This section will override anything in the system defaults or other
## sections here.
[overrides]
## output background color--only used by html and epub (and ignored in
## epub by many readers). Included below in output_css--will be
## ignored if not in output_css.
background_color: ffffff
## Allow customization of CSS. Make sure to keep at least one space
## at the start of each line and to escape % to %%. Also need
## background_color to be in the same section, if included in CSS.
## 'adobe-text-layout: optimizeSpeed;' prevents hyphenation on newer Nooks
output_css:
body { background-color: #%(background_color)s;
text-align: justify;
margin: 1%%; }
p {margin-top: 0pt; margin-bottom: 0pt; padding: 0pt; text-indent: 15pt; font-size: 1.29167em; text-align: justify;}
h1, h2, h3, h4, h5, h6 { text-align: center; padding-bottom: 10pt;}

.CI {
text-align:center;
margin-top:0px;
margin-bottom:0px;
padding:0px;
}
.center {text-align: center;}
.cover {text-align: center;}
.full {width: 100%%; }
.quarter {width: 25%%; }
.smcap {font-variant: small-caps;}
.u {text-decoration: underline;}
.bold {font-weight: bold;} .bold {font-weight: bold;}

I've attached the story in question: Here is the url: http://archiveofourown.org/works/1036957

Thank you in advance for any and all help,
Amalthia

[JimmXinu]

Originally Posted by Amalthia

Here is my personal.ini info:

Well, the simple fix is turn replace_br_with_p off. That story doesn't need it.

However, I do see that replace_br_with_p is doing some slightly odd things. I'll let Asbjorn know about it and see if he can do anything about it.

Keep in mind that replace_br_with_p is a heuristic process. It follows rules of thumb and assumptions and is never going to be 100% perfect in all cases.

[Amalthia]

Originally Posted by JimmXinu

Well, the simple fix is turn replace_br_with_p off. That story doesn't need it.

However, I do see that replace_br_with_p is doing some slightly odd things. I'll let Asbjorn know about it and see if he can do anything about it.

Keep in mind that replace_br_with_p is a heuristic process. It follows rules of thumb and assumptions and is never going to be 100% perfect in all cases.

I turned it off for now. I figured I'd save it for stories that really do use br tags.

I'll read the link you provided and see what it says. It could be that with this turned on I no longer need to specify the font, or it's being embedded twice which I know can increase the font size.

[Feynor]

Sometimes you get a problem, and you scoff, saying "That's easy".

Then 3 hours, and at least two pots of coffee later you have rewritten half your code, and added another 30%.

I'll be committing a new version of the replace_br_with_p code after a few more tests.

[Feynor]

Originally Posted by Amalthia

I turned it off for now. I figured I'd save it for stories that really do use br tags.

The next build should fix the problem, though one of the culprits is your font-size being specified in the p element of your css.

It is still larger. Without the font-size specified, the chapter text is the same size as the one in the summary page.

The attached file is made with that new build, but I replaced its css with the one from your copy.

Edit: I've added the version of the file using my own css, just for comparison.

output_css: 
 body { background-color: #%(background_color)s;
        text-align: justify;
        margin: 1em;
        adobe-hyphenate: none; }
 p { text-indent: 1em; 
        text-align: justify;
        margin-top: 0px;
        margin-bottom: 1.5ex; }
 pre { font-size: x-small; }
 sml { font-size: small; }
 h1, h2, h3, h4, h5, h6 { text-align: center; }
 .CI {
     text-align:center;
     margin-top:0px;
     margin-bottom:0px;
     padding:0px;
     }
 .center   {text-align: center;}
 .cover    {text-align: center;}
 .full     {width: 100%%; }
 .quarter  {width: 25%%; }
 .smcap    {font-variant: small-caps;}
 .u        {text-decoration: underline;}
 .bold     {font-weight: bold;}

[JimmXinu]

Attached is a test version with Asbjorn / Feynor's latest changes for replace_br_with_p.

And thanks for tackling this whole feature, I really didn't want to.

UPDATE -- replaced with slightly newer version.

UPDATE Nov 16, 2013 -- Remove obsolete beta versions

[Amalthia]

Originally Posted by Feynor

The next build should fix the problem, though one of the culprits is your font-size being specified in the p element of your css.

It is still larger. Without the font-size specified, the chapter text is the same size as the one in the summary page.

The attached file is made with that new build, but I replaced its css with the one from your copy.

Edit: I've added the version of the file using my own css, just for comparison.

Thank you for investigating this in more detail for me. I'm going to try out the newer version and tweak my settings. It appears that I may no longer need to specify the font size.

[Judith7274]

I'm almost afraid to post my question because the posts in the thread are beyond my understanding. Anyway

I have calibre. I installed fanfiction downloader as a plug in. How do I get the story into calibre and onto my kindle. I don't see what to click in Calibre to get the story. It's from fanfiction.net.

Thanks.

Thanks

[JimmXinu]

Originally Posted by Judith7274

I have calibre. I installed fanfiction downloader as a plug in. How do I get the story into calibre and onto my kindle. I don't see what to click in Calibre to get the story. It's from fanfiction.net.

As per the Introduction to plugins post, make sure that you restarted calibre after installing the plugin, and that you've added it to a menu, toolbar using Preferences->Toolbar. If you don't know what the different toolbar options are, add FanFictionDownLoader to 'the main toolbar'.

Once it's on your toolbar, click it to add a story by URL. For Kindle you can either create mobi files directly from FFDL or download to epub and then convert to mobi or azw3 files for Kindle (the preferred method).

[Judith7274]

Thanks for the quick reply. I did restart Calibre, but did it again after restarting my whole computer and the plug-in is on the toolbar now. Thanks again.

[Jester11]

I know that this is not something on your end but just a head's up that stories on Fanfiction.net that I attempt to download come back with this error:

'NoneType' object has no attribute 'find'

Just thought you would like to know.

[Ellachanted]

Hi!
I got the same error message tonight. I got it earlier today, but the latest update took care of it. Then tonight it is back.

Don't ya love FFnet?

[Feynor]

If I'm right, fixing ff.net will take a little while this time.

At a glance it looks like FF.net changed their entire summary header.

I tried debug on one of the many ff.net stories I've used to test the new heuristics.

FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(702):FanFictionDownLoader v1.7.52
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(737):url:http://www.fanfiction.net/s/293768/1/
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fanfictionnet(80):URL: http://www.fanfiction.net/s/293768/1/
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fanfictionnet(113):=Trying newer chapter: http://www.fanfiction.net/s/293768/13/
FFDL:INFO:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fanfictionnet(153):Fall back category collection
FFDL:ERROR:calibre_plugins.fanfictiondownloader_plugin.dialogs(567):Exception: {u'comment': u"'NoneType' object has no attribute 'find'", u'publisher': u'www.fanfiction.net', u'begin': None, u'good': False, u'end': None, u'author': [u'Jeffrey Vasquez'], u'url': u'http://www.fanfiction.net/s/293768/1/', u'listorder': 0, u'author_sort': u'Vasquez, Jeffrey', u'comments': u'', u'calibre_id': 3715, u'added': False, u'title': u'Progeny'}:'NoneType' object has no attribute 'find'
Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.dialogs", line 557, in do_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 791, in prep_download_loop
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter", line 243, in getStoryMetadataOnly
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fanfictionnet", line 170, in extractChapterUrlsAndMetadata
AttributeError: 'NoneType' object has no attribute 'find'

Fir Jim, here's the header code fort that one.

<hr>
<button class='btn pull-right' type=button onClick='$("#follow_area").modal();'>Follow/Favorite</button><b class='xcontrast_txt'>Progeny</b>
<span class='xcontrast_txt'><div style='height:5px'></div>By:</span> <a class='xcontrast_txt' href='/u/72474/Jeffrey-Vasquez'>Jeffrey Vasquez</a> <img align=absmiddle src='http://fast.fictionpressllc.netdna-cdn.com/static/ficons/email.png' width=16 height=16 border=0> <a class='xcontrast_txt' title="Send Private Message" href='https://www.fanfiction.net/pm2/post.php?uid=72474'>PM</a>
<div style='margin-top:2px' class='xcontrast_txt'>Memory lost. Memory Forgotten. Shadows dance under the light of the moon. Ranma Saotome thought that he was invincible. Nothing in the world could best him until someone close to his heart was taken from him.</div>
<span class='xgray xcontrast_txt'>Rated: <a class='xcontrast_txt' href='http://www.fictionratings.com/' target='rating'>Fiction T</a> - English - Drama/Romance - Ami M./Amy/Sailor Mercury, Ranma - Chapters: 12 - Words: 209,576 - Reviews: <a href='/r/293768/'>243</a> - Favs: 353 - Follows: 313 - Updated: 03-24-10 - Published: 05-24-01 - id: 293768 </span>
<div class='lc-wrapper' style='margin-top:10px'><div class='lc'>
<span class='xcontrast_txt' style='font-size:15px;cursor:pointer;' title="Increase Font Size" onClick="_fontastic_change_size('u');" >A+</span>  <span class='xcontrast_txt' style='font-size:13px;cursor:pointer;' title="Decrease Font Size" onClick="_fontastic_change_size('d');" >A-</span>  <img onclick="_fontastic_init('reading');$('#_fontastic_reading').modal('show');" style='margin-left:2px;margin-right:2px;' align=absmiddle src='http://fast.fictionpressllc.netdna-cdn.com/static/fcons/edit-style.png' width=16 height=16 border=0 title="Fonts">
<img onclick="$('#f_width').slideToggle();" style='margin-left:2px;margin-right:2px;' align=absmiddle src='http://fast.fictionpressllc.netdna-cdn.com/static/fcons/edit-alignment-justify.png' width=16 height=16 border=0 title="Story Width"> <span id=f_width class=zhide> <a href='#' onclick='_fontastic_change_width(100);'>Full</a> <a href='#' onclick='_fontastic_change_width(75);'>3/4</a> <a href='#' onclick='_fontastic_change_width(50);'>1/2</a></span> <img onclick="$('#f_size').slideToggle();" style='margin-left:2px;margin-right:2px;' align=absmiddle src='http://fast.fictionpressllc.netdna-cdn.com/static/fcons/edit-line-spacing.png' width=16 height=16 border=0 title="Story Spacing"> <span id=f_size class=zhide> <a href='#' onclick='_fontastic_change_line_height("u");'>Expand</a> <a href='#' onclick='_fontastic_change_line_height("d");'>Tighten</a></span> <img onclick="toggleTheme();" style='margin-left:2px;margin-right:2px;' align=absmiddle src='http://fast.fictionpressllc.netdna-cdn.com/static/fcons/contrast.png' width=16 height=16 border=0 title="Story Contrast"></div></div>
<span style='float:right; ' > <SELECT id=chap_select title="Chapter Navigation" Name=chapter onChange="self.location = '/s/293768/'+ this.options[this.selectedIndex].value + '/Progeny';"><option value=1 selected>1. Progeny Chapter 1<option value=2 >2. Progeny Chapter 2<option value=3 >3. Progeny Chapter 3<option value=4 >4. Progeny Chapter 3: Part II<option value=5 >5. Progeny Chapter 4<option value=6 >6. Progeny Chapter 5<option value=7 >7. Chapter 6<option value=8 >8. Chapter 7<option value=9 >9. Chapter 8<option value=10 >10. Chapter 9<option value=11 >11. Chapter 10<option value=12 >12. Chapter 11</select> <button class=btn TYPE=BUTTON onClick="self.location='/s/293768/2/Progeny'">Next &gt;</button></span><div style='height:5px'></div><script>

[Feynor]

Originally Posted by Ellachanted

Hi!
I got the same error message tonight. I got it earlier today, but the latest update took care of it. Then tonight it is back.

Don't ya love FFnet?

Actually, if you used "replace_br_with_p" that was my error earlier.

[JimmXinu]

New test version with fix for latest ff.net change.

I haven't tested comprehensively, but it appears to work. I have to go now, but let me know if any of the metadata isn't coming through correctly.

UPDATE Nov 16, 2013 -- Remove obsolete beta versions