[GUI Plugin] FanFicFare

[JimmXinu]

New Test Version Posted

2025-04-11
- adapter_fictionmaniatv: Update for ancient stories

Arguably, we could declare those stories flawed.

But there's still more brute force we can try first!

Adapter now looks for a beginning HTML comment and an ending comment (and <hr>) and blindly assumes if both of those markers are found, whatever is between them is the chapter text. Which works in the relatively few number of cases I've tried.

[JimmXinu]

New Test Version Posted

2025-04-12
- adapter_ficbooknet: Add series collection & fix downloads, thanks dbhmw

[icecold]

How long should gathering metadata take for a single ao3 fic nowadays? FFF gets stuck on that step with ao3 fics now.

Edit: Never mind, I found that bit of the debug log for the one time FFF didn't get stuck and downloaded an ao3 fic properly and the answer was within an hour. Probably much quicker but I can't parse most of the debug log.

[ninjadudedxp]

I found some old copies of RR books, i took the ones with older chapters and added the new Chapters also changing the HTML names to line up with the chapter order, should those books be ok when a new chapter gets added? or will it update based on what was there last and overwrite the next file that was suppose to be in the sequence rather than checking what chapters are available .

[JimmXinu]

Multi-reply

Originally Posted by icecold

How long should gathering metadata take for a single ao3 fic nowadays? FFF gets stuck on that step with ao3 fics now.

Edit: Never mind, I found that bit of the debug log for the one time FFF didn't get stuck and downloaded an ao3 fic properly and the answer was within an hour. Probably much quicker but I can't parse most of the debug log.

AO3 metadata collection is only a couple page requests. If it's taking that long, it's because of blocking, or the ongoing AO3 server issues.

Many of us have switched either using FFF's Browser Cache mode or the new use_archive_transformativeworks_org setting. These have been discussed before, search the thread for more.

Originally Posted by ninjadudedxp

I found some old copies of RR books, i took the ones with older chapters and added the new Chapters also changing the HTML names to line up with the chapter order, should those books be ok when a new chapter gets added? or will it update based on what was there last and overwrite the next file that was suppose to be in the sequence rather than checking what chapters are available .

If you got everything exactly correct, including FFF's embedded chapter URL, maybe? If any chapters have the embedded <meta> tag for chapter URL, FFF will expect *all* chapter files to have it and discard any that don't.

If you are just trying to save download time, you could go ahead and try it and the worst that happens is chapters that aren't correct get downloaded again.

OTOH, if you are trying to save 'stubbed'/removed chapters or something like that, I would suggest setting your manually edited version aside as separate book (clear the identifiers:url field) and use FFF to download a current copy in a different book.

[iakrus]

hi!
i havent had access to my computer (and thus my calibre library) for a few months, and i just got it going again. i updated to version 8 and updated all my plugins. I tried to add/update some ao3 fanfics and discovered the issues.
Can someone give me a quick rundown on the problem/solutions?

As a sidenote, I was wondering if anyone knows what plugin allowed me to add metadata under my kobo readers fic titles (as pictured below). That has also broken with the updating of the plugins, presumably because V8 was kobo update heavy? I know this isnt the right thread for this, but I figured I'd try and solve two issues with one post- I'll repost elsewhere if no one here knows!

thanks!

[icecold]

It getting stuck was with me using .archive subdomain. Which I started using because using the browser cache doesn't seem to work. Even with doing everything that's been discussed in thread getting a fic from ao3 to download properly just seems like luck.I didn't change my .ini between the one successful download and the attempts that got stuck.

[watani]

Seems that Spacebattles has done a website update sometime over the past few hours because I immediately get a login prompt from FFF when I try to download any thread (which always fails even when the correct credentials are given with a 403 forbidden on the Spacebattles login page) and when I try to use the browser cache I get an error saying: 'NoneType' object has no attribute 'get_text'.

Edit: I do not have a problem with Sufficientvelocity threads, and I have no problem accessing Spacebattles through my browser.

[ninjadudedxp]

Originally Posted by JimmXinu

Multi-reply


AO3 metadata collection is only a couple page requests. If it's taking that long, it's because of blocking, or the ongoing AO3 server issues.

Many of us have switched either using FFF's Browser Cache mode or the new use_archive_transformativeworks_org setting. These have been discussed before, search the thread for more.


If you got everything exactly correct, including FFF's embedded chapter URL, maybe? If any chapters have the embedded <meta> tag for chapter URL, FFF will expect *all* chapter files to have it and discard any that don't.

If you are just trying to save download time, you could go ahead and try it and the worst that happens is chapters that aren't correct get downloaded again.

OTOH, if you are trying to save 'stubbed'/removed chapters or something like that, I would suggest setting your manually edited version aside as separate book (clear the identifiers:url field) and use FFF to download a current copy in a different book.

Yes I have version prestubbed before I changed my set up. Wanted to merge them. I'll just keep them separate thanks for the input

[ninjadudedxp]

Edit: Cant seem to edit my own post, but the way i pulled the chapters was with Calibre Epub edit and i exported Each HTML , i think they are Embeded with the FFF link in them as i see the body class showing it.

[FlyingSpaceOxen]

Originally Posted by JimmXinu

Arguably, we could declare those stories flawed.

But there's still more brute force we can try first!

Adapter now looks for a beginning HTML comment and an ending comment (and <hr>) and blindly assumes if both of those markers are found, whatever is between them is the chapter text. Which works in the relatively few number of cases I've tried.

Thanks! So far, so good. Downloaded SWI stories now largely appear as they were intended, awkward original image placement by the authors notwithstanding in some entries.

Somewhat unrelated: I've noticed the download_text_version option and read the original author's comment in the extractor code explaining their use of line breaks to preserve 1:1 the presentation of text stories on FictionMania. However this results in EPUBs with non-reflowable text. Using the default settings, FanFicFare produces EPUBs that ignore multiple new lines that are used, for example, to separate scenes or chapters in a single text file unless the author uses something like a Markdown-compatible horizontal rule.

I imagine this is a HTML limitation, correct? Up till now, I've been taking the plain text files provided by FictionMania, turning them into properly formatted Markdown files, then running them through Pandoc to generate EPUB files that faithfully preserve their original presentation.

[JimmXinu]

Multi-reply

Originally Posted by watani

Seems that Spacebattles has done a website update sometime over the past few hours ...

I'm not seeing this. Unless other people are seeing it happen, I suspect it's you/your IP/network being throttled or blocked.

Originally Posted by FlyingSpaceOxen

Somewhat unrelated: I've noticed the download_text_version option and read the original author's comment in the extractor code explaining their use of line breaks to preserve 1:1 the presentation of text stories on FictionMania. However this results in EPUBs with non-reflowable text.

Honestly, the download_text_version option is just for historical back-compat. The only particularly good use case I see for it would be someone outputting to text format. If you want flowable output, don't use download_text_version.

[watani]

If that's the case where I'm getting throttled/blocked, then why is it that when I test a different IP using a VPN I still am getting the same issue? And why is it that I have no issue accessing the site via a browser?

Regardless, I also am unable to use the browser cache to get past the issue. Running in debug mode has the log say it's unable to decode the first byte of the cached page.

[JimmXinu]

Cloudflare is a large business devoting many resources to detecting and blocking 'undesired' traffic and many large sites use it. I assume they have all the same tracking capabilities as the major ad platforms, and can detect the difference between an automated program like FFF and a real browser. Were you doing a lot of downloads/updates when the problem started?

I've just now downloaded another story update from SB without problems, so I continue to believe it's not a universal issue.

I don't believe SB/etc are sites I've put any effort into testing BrowserCache with. A quick look now shows page response from SB marked no-cache by the server, so I wouldn't expect it to work.

[watani]

No, I was not doing any large number of downloads/updates when the problem started. I had successfully downloaded a thread about 6 hours prior and went to bed, and when I woke up and tried to download threads I got the errors.

If I can't download via FFF, then what about using a different tool and then updating the FFF epub? I've had success using WebToEpub, but it's not clear that there's a way to easily merge in the new chapters to match the FFF file-naming structure automatically.