[GUI Plugin] FanFicFare

[amythyr]

I always download 10 stories at a time , after i read all of them(~5-6 days) then download 10 again.
Chapter numbers vary, some have 5 or 6 and some have 30-40 chapters, but approximate word count is 200k

Even if i download single link it is taking time.

Sorry for late reply, i only remembered when i got link i email.

[JimmXinu]

Originally Posted by amythyr

I always download 10 stories at a time , after i read all of them(~5-6 days) then download 10 again.
Chapter numbers vary, some have 5 or 6 and some have 30-40 chapters, but approximate word count is 200k

Even if i download single link it is taking time.

ffnet downloads take at least ~8-10 seconds per page request (chapter). So a 30 chapter story should take ~5-6 minutes at best.

Because FFF will retry several times (with increasingly longer waits between tries) if it has connection failures, it can take longer. Sometimes a lot longer.

As long as the downloads succeed eventually, FFF is working correctly. There are many possible reasons why you may be seeing slow downloads. Pretty much all of them are outside our control.

[saluli]

A wiki page with the suggestions of how people are using the replace metadata would be great. I must confess I always look when someone posts theirs personal.ini to get new ideas.

I had problems with error messages when trying to do a lot alterations at the same time. If I am not mistaken the error was 'too many values to unpack', and it mostly happened when I tried some complex rule changing a lot of things per line, or added to much changes at same time. Always a pain to discover which line was giving problem, the plugin always marked the whole section as error.

[JimmXinu]

New test version posted (direct link)https://www.mobileread.com/forums/att...8&d=1446846585 that improves error reporting of lengthy personal.ini lines.

[saluli]

Tomorrow I will look in my old txt folder and test some of my problematic regex.

Meanwhile, it is possible to customize how the Publisher is filled for each site? I want to use the name of the archives instead of the url....

[JimmXinu]

Originally Posted by saluli

Meanwhile, it is possible to customize how the Publisher is filled for each site? I want to use the name of the archives instead of the url....

Yes, it is. What ends up in calibre's Publisher field (and labeled Publisher in the title page) is actually the site metadata, which you can change the usual way:

[defaults]
add_to_replace_metadata:
 site=>archiveofourown.org=>AO3
 site=>test1.com=>Test One

[Firedancer885]

Jimm,

I just tried to drag and drop a link from Chrome into the download window and got this error.

Spoiler:

calibre, version 2.42.0
HIBA: Ismeretlen hiba történt: <b>UnicodeDecodeError</b>:'utf8' codec can't decode byte 0xe9 in position 84: invalid continuation byte

calibre 2.42  isfrozen: True is64bit: True
Linux-3.19.0-30-generic-x86_64-with-debian-jessie-sid Linux ('64bit', 'ELF')
('Linux', '3.19.0-30-generic', '#34-Ubuntu SMP Fri Oct 2 22:08:41 UTC 2015')
Python 2.7.9
Linux: ('debian', 'jessie/sid', '')
Successfully initialized third party plugins: Antikvarium_hu && Count Pages && Reading List && FanFicFare && EpubSplit && Generate Cover && EpubMerge && djvumaker && View Manager && Libri_hu
Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.dialogs", line 202, in dropEvent
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 84: invalid continuation byte

Here is the link to the story. http://archiveofourown.org/works/4670174

I'm using Ubuntu 15.10 64bit. I asked about this before. I have attached a screenshot about the possible cause. In the shot I'm dragging the link.

[JimmXinu]

Originally Posted by Firedancer885

I just tried to drag and drop a link from Chrome into the download window and got this error.

Spoiler:

calibre, version 2.42.0
HIBA: Ismeretlen hiba történt: <b>UnicodeDecodeError</b>:'utf8' codec can't decode byte 0xe9 in position 84: invalid continuation byte

calibre 2.42  isfrozen: True is64bit: True
Linux-3.19.0-30-generic-x86_64-with-debian-jessie-sid Linux ('64bit', 'ELF')
('Linux', '3.19.0-30-generic', '#34-Ubuntu SMP Fri Oct 2 22:08:41 UTC 2015')
Python 2.7.9
Linux: ('debian', 'jessie/sid', '')
Successfully initialized third party plugins: Antikvarium_hu && Count Pages && Reading List && FanFicFare && EpubSplit && Generate Cover && EpubMerge && djvumaker && View Manager && Libri_hu
Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.dialogs", line 202, in dropEvent
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 84: invalid continuation byte

Here is the link to the story. http://archiveofourown.org/works/4670174

I'm using Ubuntu 15.10 64bit. I asked about this before. I have attached a screenshot about the possible cause. In the shot I'm dragging the link.

The last time was in July. Is this something you're seeing a lot, or just once in a great while? Because what I said then still applies:

Originally Posted by JimmXinu

I was able to duplicate the issue with a live-cd linux desktop before the entire OS crashed. Twice. And I didn't find a current calibre distro for Raspberry Pi, the other linux-ish hardware I have handy.

But I was able to confirm that other URLs drag and dropped fine. So I'm blaming the unicode chars in the title: hoʻokāne.

The drag and drop event object hands me data in a 'QByteArray' object. I don't know why Chrome on linux is (presumably) including the link text along with the URL, but that's what I speculate is happening.

If you bump into it a lot, I might look at it some more. But as it is, I'm inclined to call it an odd corner case and not worry about it.

[edeniz]

Something strange I just noticed: When downloading this story:

http://archiveofourown.org/works/4766726

as an epub, the file does not get any sort of table of contents. It has seven chapters. The html of the same fic works perfectly fine as far as I can see. I have not noticed something similar happening in other epubs, be it from AO3 or elsewhere.

Would someone please try to download the story themselves to see if it's just me (if so, how?) or if it's got to do with the story itself? Thank you!

I should clarify: I opened the book on calibre's in-built ebook reader. It is there I noticed this. I have not yet uploaded the fic to my reader.

[JimmXinu]

Originally Posted by edeniz

Something strange I just noticed: When downloading this story:

Looks perfectly normal to me.

The TOC for epub is in metadata, so unless you click the 'TOC' button in the calibre viewer, you won't see it.

ADDED: Post your epub and I'll look at it if you still think there's a problem.

[edeniz]

Originally Posted by JimmXinu

Looks perfectly normal to me.

unless you click the 'TOC' button in the calibre viewer, you won't see it.

That's exactly how I noticed that there wasn't a TOC to see. Ran a check with calibre's editor, but it won't find a problem. And in the meantime I tried to upload the fic to my reader, and it would not work.

Uploaded the epub in question.

[JimmXinu]

Originally Posted by edeniz

Uploaded the epub in question.

Assuming you're still using the personal.ini that uploaded earlier, the problem is because you're using chapter_title_strip_pattern to remove chapters titles like 'Chapter 1', but you didn't set add_chapter_numbers true, so nothing is replacing them.

Relevant settings:

Spoiler:

## Do not like the numbers at the start of chapter titles on some
## sites?  You can use strip_chapter_numbers to strip them off.  Just
## want to make them all look the same?  Strip them off, then add them
## back on with add_chapter_numbers:true.  Only want them added back
## on for Table of Contents(toc)? Use add_chapter_numbers:toconly.
## (toconly does not work on mobi output.)  Do not like the way it
## strips numbers or adds them back?  See chapter_title_strip_pattern
## and chapter_title_add_pattern.
strip_chapter_numbers:true

## add_chapter_numbers can be true, false or toconly
## (Note number is not added when there is only one chapter.)
##add_chapter_numbers:true

## (Two versions of chapter_title_strip_pattern are shown below.  You
## should only have one uncommented.)
## This version will remove the leading number from:
## "1." => ""
## "1. The Beginning" => "The Beginning"
## "1: Start" => "Start"
## "2, Chapter the second" => "Chapter the second"
## etc
#chapter_title_strip_pattern:^[0-9]+[\.: -]+

## This version will strip all of the above *plus* remove 'Chapter 1':
## "Chapter 1" => ""
## "1. Chapter 1" => ""
## "1. Chapter 1, Bob's First Clue" => "Bob's First Clue"
## "Chapter 2 - Pirates Place" => "Pirates Place"
## etc
chapter_title_strip_pattern:^([0-9]+[\.: -]+)?(Chapter *[0-9]+[\.:, -]*)?

You could add add_chapter_numbers:true and you'd get chapters '1.', '2.', etc.

You could set strip_chapter_numbers:false or change to the default chapter_title_strip_pattern instead of the more aggressive one.

Alternatively, you could experiment with look ahead regex pattern matches for chapter_title_strip_pattern so you only remove 'Chapter 1' if there's something after it.

I will also consider if I should change the system in some way to prevent this behavior, or if it would ever be desirable.

[edeniz]

Originally Posted by JimmXinu

Assuming you're still using the personal.ini that uploaded earlier, the problem is because you're using chapter_title_strip_pattern to remove chapters titles like 'Chapter 1', but you didn't set add_chapter_numbers true, so nothing is replacing them.

Alternatively, you could experiment with look ahead regex pattern matches for chapter_title_strip_pattern so you only remove 'Chapter 1' if there's something after it.

I will also consider if I should change the system in some way to prevent this behavior, or if it would ever be desirable.

Nah, you don't need to change the system when it works perfectly fine otherwise.(At least not because of my incompetence.)

But, since I'm really bad at regex, I think I'll just have to use the less aggressive stripping string.

But there is something else wrong with this epub that calibre's epub editor can't find. Specifically, the... probably the metadata of the titlepage entries? This is what happened: Even when I use the other stripping pattern, or use no stripping at all, it does not matter. The epub will not be added the my reader's library. In response, I deleted the epub and simply let calibre convert to epub from the html file of the same fic. I let calibre add its default cover page to the fic. This epub was added to my reader's library without problems. But when I opened it, I noticed I had to immediately skip to a page or chapter after the cover page, skipping the titlepage entirely, or it would basically crash and close the fic or freeze up. It happens with both my readers and my smartphone.

Basically, whether I convert the html file to epub via calibre, or I download the file from ao3 as epub, the fic will cause problems that I cannot find the root of.

Do you have any idea what could be the reason?

[JimmXinu]

Originally Posted by edeniz

...
Basically, whether I convert the html file to epub via calibre, or I download the file from ao3 as epub, the fic will cause problems that I cannot find the root of.

Do you have any idea what could be the reason?

That suggests either something wrong with the story content or that you're not actually replacing the previous content when you think you are.

I've tried it in a few different readers and haven't had any problems with the FFF generated epub.

You could try changing the title before sending it to your device and see if it works better. The device might be caching some bad data.

The only unusual thing I found in the content was this from chapter 7:

Spoiler:

<p><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">Walking</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">the</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">streets</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">seemed</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">even</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">more</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">a</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">solace</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">that</a><a href="https://docs.google.com/document/d/1JzCppa4YIDjcRgU0rQnYdwzn8pfhLS1UqU3kGQiYYeo/edit?usp=sharing">night</a>.</p>
<p>##</p>

[edeniz]

OK. Since you said you had no trouble with different readers, I tried it this way: I downloaded the fic with an empty personal.ini. It worked on all three of my devices without problems. So, obviously, I messed up something there. Compared it to defaults, but couldn't see anything. To hunt the error down more easily, I trimmed personal.ini down to only the necessities, but I still can't find it. I've attached it. Can you see anything that doesn't belong there and causes this error? If it isn't too much trouble I mean?