[GUI Plugin] FanFicFare - Page 447

icecold · 11-13-2021, 12:04 PM

I've been having an issue with how I've been adding the word count and date posted to SB/SV/QQ with custom output and I'm pretty sure that it's mostly a mistake on my end that's causing it. I think that when I'm updating a story it for some reason re-adding the word count and date.

Spoiler:

Code:

[base_xenforoforum:epub]
## base_xenforoforum stories with threadmarks have a few additional
## pieces of metadata available that most adapters don't. 'date';
## 'words'--word count as number only: "12104"; and kwords--the
## threadmark human-readable string for word count with parens:
## "(12.1k)" These can be used with custom output (see
## https://github.com/JimmXinu/FanFicFare/wiki/CustomOutput ) or with
## chapter_title_*_pattern settings.
## Examples for epub:
#[base_xenforoforum:epub]
tocpage_entry:
 <a href="file${index04}.xhtml">${chapter}</a> Posted: ${date} Word Count: ${kwords}<br /><br />
tocpage_start:
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <title>${title} by ${author}</title>
 <link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/>
 </head>
 <body class="fff_tocpage">
 <div>
 <h3 class="tocpage-notes">Notes</h3>
 <li>This is probably best read with dark mode. Quotes and Spoilers probably won't be easily readable</li>
 <li>Date posted is in mm/dd/yyyy format.</li>
 <hr>
 <h3>Table of Contents</h3>

chapter_start:<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <title>${chapter}</title>
 <link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/>
 <meta name="chapterurl" content="${url}"></meta>
 <meta name="chapterorigtitle" content="${origchapter}"></meta>
 <meta name="chaptertoctitle" content="${tocchapter}"></meta>
 <meta name="chaptertitle" content="${chapter}"></meta>
 </head>
 <body class="fff_chapter">
 <h4 style="text-align: right; margin: 0px;">Posted: ${date} <br>Word Count: ${kwords}</h4><br/>
 <h3 class="fff_chapter_title">${chapter}</h3>

## The 'date' value for chapters mentioned above can be formated with
## datethreadmark_format.  Otherwise it will default to
## dateCreated_format
#datethreadmark_format:%%Y-%%m-%%d %%H:%%M
datethreadmark_format:%%m/%%d/%%Y

Worst case scenario I just stop inserting kwords and date in every chapter and just rely on the toc.

JimmXinu · 11-13-2021, 01:19 PM

By default, when reusing a chapter text, FFF removes the first h3 (and h2) tag(s) found and keeps everything else in the body.

It also will remove all tags with the class skip_on_ffdl_update for just this sort of occasion.

So add class="skip_on_ffdl_update" on your <h4> tag and it should work as desired in future. That won't help clean up existing chapters though. Also, I'd move the last <br> inside the <h4> so you don't end up with an ever growing vertical space.

It looks like that feature isn't mentioned in custom output wiki page. I will add it.

Quote:

Originally Posted by icecold

I've been having an issue with how I've been adding the word count and date posted to SB/SV/QQ with custom output and I'm pretty sure that it's mostly a mistake on my end that's causing it. I think that when I'm updating a story it for some reason re-adding the word count and date.

JimmXinu · 11-14-2021, 01:03 PM

New Test Version Posted

2021-11-14
- New Site: psychfic.com (re-added), thanks HappyFaceSpider Closes #764

bpothier · 11-17-2021, 02:27 AM

This isn't strictly a FFF issue, but Calibre seems to take a *LOOOOONG* time when it gets to the updating books stage - after the Yes/No/View Log part.

I recently had FFF update 2000 books using the cached metadata field. It took under an hour for FFF to do its part, but Calibre took close to 20 hours to import those updated books. This is on Win10, and looking at resmon, Calibre doesn't seem to be doing much during this stage - no heavy CPU, disk or network activity. It often goes "Not Responding" and then resumes. Are there any suggested tweaks to speed up this part of things?

One "small" thing, still running Calibre 64bit 4.23.0 - no real reason not to jump to 5.x at this point, was just holding off initially for plugins to catch up...

JimmXinu · 11-17-2021, 11:08 AM

Quote:

Originally Posted by bpothier

This isn't strictly a FFF issue, but Calibre seems to take a *LOOOOONG* time when it gets to the updating books stage - after the Yes/No/View Log part.

I recently had FFF update 2000 books using the cached metadata field. It took under an hour for FFF to do its part, but Calibre took close to 20 hours to import those updated books. This is on Win10, and looking at resmon, Calibre doesn't seem to be doing much during this stage - no heavy CPU, disk or network activity. It often goes "Not Responding" and then resumes. Are there any suggested tweaks to speed up this part of things?

One "small" thing, still running Calibre 64bit 4.23.0 - no real reason not to jump to 5.x at this point, was just holding off initially for plugins to catch up...

I can't say I've seen this, but then I neither update metadata on 2000 books at a time, nor run Calibre v4.

In years past, I would speculate that doing such a large update may have exceeded your computer's working memory and it had to use virtual memory on disk which is a lot slower; I want to say 10-100x slower. However, today's computers generally have so much memory, I don't know if that's still a concern--I don't think my current computer even has virtual memory configured.

To be perfectly honest, I'm not going to spend time on a possible issue with huge mass update on an outdated version of Calibre.

JimmXinu · 11-17-2021, 01:57 PM

New Test Version Posted

2021-11-17
- New Site: www.deviantart.com, thanks jcotton42 (see #765 and #374)

bpothier · 11-17-2021, 03:03 PM

Thank you for reply, I was hoping for a quick "Oh, you might have option XX enabled" or something... no need for deep dive

I did poke around in Calibre preferences, and under "Saving books to disk" there is an option to "Update metadata in saved copies" option which defaults to enabled (and says "Makes saving to disk slower", so sounds promising). Wondering if this is doing "double updates" where FFF creates an updated ePub which gets imported to Calibre, which then goes back and updates the freshly imported ePub with the Metadata? I'll poke at it a little and will likely update to 5.x at some point and compare...

bpothier · 11-17-2021, 04:18 PM

UPATE: I tried updating/adding 14 ePUB with FFF. Once FFF completed and got to the Yes/No/Log screen, it took about 11 mins for Calibre to add/update them into the library. I then used the FFF update from stored metadata column - which took 20-30 secs - and then from the Yes/No/Log step it took 8m20s for Calibre update. I disabled that "Update metadata in saved copies" setting (which may only apply to export eBook action in Calibre) and it took the same 8m20s to update those 14 after FFF did the update from saved metadata. Going to try creating a new library in Calibre with only those 14 to compare times...

UPDATE2: From a blank Library with only those 14, it took 18s to do that Calibre update step...

ilovejedd · 11-17-2021, 04:53 PM

Quote:

Originally Posted by bpothier

This isn't strictly a FFF issue, but Calibre seems to take a *LOOOOONG* time when it gets to the updating books stage - after the Yes/No/View Log part.

I recently had FFF update 2000 books using the cached metadata field. It took under an hour for FFF to do its part, but Calibre took close to 20 hours to import those updated books. This is on Win10, and looking at resmon, Calibre doesn't seem to be doing much during this stage - no heavy CPU, disk or network activity. It often goes "Not Responding" and then resumes. Are there any suggested tweaks to speed up this part of things?

One "small" thing, still running Calibre 64bit 4.23.0 - no real reason not to jump to 5.x at this point, was just holding off initially for plugins to catch up...

Hardware specs?

I did an import test one time on various storage media. The import was unbearably slow on my netbook HDD (10x slower compared to a Samsung 830 SSD).

I eventually learned the 2.5" 7.5mm thick HDD that was pre-installed on the netbook was using Shingled Magnetic Recording (SMR). I think all those small database writes were too much for the poor drive.

Also, try vacuuming your Calibre database. It might have gotten bloated.

P.S. How many books do you have in the library? If you've got like tens or hundreds of thousands, the performance drop can be understandable.

us3rname · 11-17-2021, 05:46 PM

please help can't download these fics from asianfanfic AFF
keep getting error

Status
Error

Title Author
Empress essie_lewis03

Comment
HTTP Error in FFF '404 Client Error: Not Found for url: [redacted]

URL
[redacted]

Status
Title
Author
Comment
URL
Error
Girl-on-Girl Futanari One-Shots
JamieStardust
'NoneType' object has no attribute 'find_all'
[redacted]

JimmXinu · 11-17-2021, 06:11 PM

While I do prefer to get problem reports publicly, please either use links to SFW stories or use PMs for the NSFW links. I have removed them for you.

That said, I think you're running into site blocking. I tried and got blocked after several very quick requests.

Try adding this to your personal.ini:

Code:

[www.asianfanfics.com]
slow_down_sleep_time:2

You may need to wait minutes/hours/days for the blocking to time out, but a 2 second sleep seems to be working for me. Aside from ch79 of the longer one failing due to "This is an early access chapter for crowdfund supporters that will be released for public view in
X days, XX minutes"

If that works for you after a day or 2, try a higher number and report back what works so I can set the default, thanks.

Rellwood · 11-17-2021, 06:33 PM

Hi, I haven't scrolled over the whole thread, but I had some questions/requests.

How can I change the format of the dates in the details page to the same that I have in Calibre dd MMM yyyy

Is there a way of updating the details page - after metadata changes have occured in the edit metadata? I want to include the ENF plugin's column AFTER I run it after the book has been downloaded. As of now, what I have been doing is copying over the column into the comments field (I don't like the way it's formatted so I have it in the way I like) and then the comments field gets added in the book jacket - but this isn't the details page. Maybe adding columns that aren't indicative to the website, but are Calibre custom columns that I would like to see in the details page.

After the book has been created, then all the metadata updating that would be done by the user outside of the plugin can be done, then the plugin can update the details page?

I would like the ability to "re-run" the update metadata to include new items in the details page, but not have to use the plugin to go to the site in case the story has been removed.

bpothier · 11-17-2021, 06:50 PM

UPDATE3: TLDR - "fixed" DB performance issue... Library & DB were on NAS, copied metadata.db to local drive and set ENV VAR "CALIBRE_OVERRIDE_DATABASE_PATH" to point to local file.... metadata update of 14 ePUB dropped to 5 seconds.

Ref: https://www.mobileread.com/forums/sh...d.php?t=294998

Hopefully this helps someone else with DB perf issues...
Details:

Spoiler:

JimmXinu · 11-17-2021, 08:33 PM

Quote:

Originally Posted by Rellwood

How can I change the format of the dates in the details page to the same that I have in Calibre dd MMM yyyy

From defaults.ini:

Code:

## Date formats used by FanFicFare.  Published and Update don't have time.
## See http://docs.python.org/library/datetime.html#strftime-strptime-behavior
## Note that ini format requires % to be escaped as %%.
dateCreated_format:%%Y-%%m-%%d %%H:%%M:%%S
datePublished_format:%%Y-%%m-%%d
dateUpdated_format:%%Y-%%m-%%d

Some sites override these when they have times for published and updated, so I'd put these under [epub] or even [overrides] to take precedence over site level defaults.

Quote:

Originally Posted by Rellwood

Is there a way of updating the details page - after metadata changes have occured in the edit metadata?
...
I would like the ability to "re-run" the update metadata to include new items in the details page, but not have to use the plugin to go to the site in case the story has been removed.

Use Update Always mode.

I spent a fair amount of time a couple different times trying to fit that ability in without re-architecting or being completely non-intuitive and gave up. Past post.

You might consider using Calibre's metadata jacket--it can be applied to any book, not just FFF output, although has other complications. Past post.

ilovejedd · 11-18-2021, 01:15 AM

Quote:

Originally Posted by bpothier

UPDATE3: TLDR - "fixed" DB performance issue... Library & DB were on NAS, copied metadata.db to local drive and set ENV VAR "CALIBRE_OVERRIDE_DATABASE_PATH" to point to local file.... metadata update of 14 ePUB dropped to 5 seconds.

Ref: https://www.mobileread.com/forums/sh...d.php?t=294998

Hopefully this helps someone else with DB perf issues...
Details:

Spoiler:

Ah. Putting the library on a NAS is one of the things that's highly recommended one NOT do.

https://manual.calibre-ebook.com/faq...rked-drive-nas

Spoiler:

11-17-2021, 02:27 AM	#6694
bpothier Member Posts: 20 Karma: 10 Join Date: May 2020 Device: Kobo	This isn't strictly a FFF issue, but Calibre seems to take a LOOOOONG time when it gets to the updating books stage - after the Yes/No/View Log part. I recently had FFF update 2000 books using the cached metadata field. It took under an hour for FFF to do its part, but Calibre took close to 20 hours to import those updated books. This is on Win10, and looking at resmon, Calibre doesn't seem to be doing much during this stage - no heavy CPU, disk or network activity. It often goes "Not Responding" and then resumes. Are there any suggested tweaks to speed up this part of things? One "small" thing, still running Calibre 64bit 4.23.0 - no real reason not to jump to 5.x at this point, was just holding off initially for plugins to catch up... Last edited by bpothier; 11-17-2021 at 02:31 AM.

11-17-2021, 04:18 PM	#6698
bpothier Member Posts: 20 Karma: 10 Join Date: May 2020 Device: Kobo	UPATE: I tried updating/adding 14 ePUB with FFF. Once FFF completed and got to the Yes/No/Log screen, it took about 11 mins for Calibre to add/update them into the library. I then used the FFF update from stored metadata column - which took 20-30 secs - and then from the Yes/No/Log step it took 8m20s for Calibre update. I disabled that "Update metadata in saved copies" setting (which may only apply to export eBook action in Calibre) and it took the same 8m20s to update those 14 after FFF did the update from saved metadata. Going to try creating a new library in Calibre with only those 14 to compare times... UPDATE2: From a blank Library with only those 14, it took 18s to do that Calibre update step... Last edited by bpothier; 11-17-2021 at 04:22 PM.

11-17-2021, 05:46 PM	#6700
us3rname Junior Member Posts: 1 Karma: 10 Join Date: Sep 2021 Device: none	please help can't download these fics from asianfanfic AFF keep getting error Status Error Title Author Empress essie_lewis03 Comment HTTP Error in FFF '404 Client Error: Not Found for url: [redacted] URL [redacted] Status Title Author Comment URL Error Girl-on-Girl Futanari One-Shots JamieStardust 'NoneType' object has no attribute 'find_all' [redacted] Last edited by JimmXinu; 11-17-2021 at 06:03 PM. Reason: Removed NSFW links

11-17-2021, 06:11 PM	#6701
JimmXinu Plugin Developer Posts: 6,981 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	While I do prefer to get problem reports publicly, please either use links to SFW stories or use PMs for the NSFW links. I have removed them for you. That said, I think you're running into site blocking. I tried and got blocked after several very quick requests. Try adding this to your personal.ini: Code: [www.asianfanfics.com] slow_down_sleep_time:2 You may need to wait minutes/hours/days for the blocking to time out, but a 2 second sleep seems to be working for me. Aside from ch79 of the longer one failing due to "This is an early access chapter for crowdfund supporters that will be released for public view in X days, XX minutes" If that works for you after a day or 2, try a higher number and report back what works so I can set the default, thanks.

11-17-2021, 06:33 PM	#6702
Rellwood Library Breeder (She/Her) Posts: 1,269 Karma: 1937891 Join Date: Apr 2015 Location: Fullerton, California Device: Paperwhite 2015 (2), PW 2024 (12 GEN), PW 2023 (11 GEN), Scribe (1st)	Hi, I haven't scrolled over the whole thread, but I had some questions/requests. How can I change the format of the dates in the details page to the same that I have in Calibre dd MMM yyyy Is there a way of updating the details page - after metadata changes have occured in the edit metadata? I want to include the ENF plugin's column AFTER I run it after the book has been downloaded. As of now, what I have been doing is copying over the column into the comments field (I don't like the way it's formatted so I have it in the way I like) and then the comments field gets added in the book jacket - but this isn't the details page. Maybe adding columns that aren't indicative to the website, but are Calibre custom columns that I would like to see in the details page. After the book has been created, then all the metadata updating that would be done by the user outside of the plugin can be done, then the plugin can update the details page? I would like the ability to "re-run" the update metadata to include new items in the details page, but not have to use the plugin to go to the site in case the story has been removed. Last edited by Rellwood; 11-17-2021 at 06:36 PM.

11-14-2021, 01:03 PM	#6693
JimmXinu Plugin Developer Posts: 6,981 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	New Test Version Posted 2021-11-14 - New Site: psychfic.com (re-added), thanks HappyFaceSpider Closes #764

11-17-2021, 01:57 PM	#6696
JimmXinu Plugin Developer Posts: 6,981 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	New Test Version Posted 2021-11-17 - New Site: www.deviantart.com, thanks jcotton42 (see #765 and #374)

11-17-2021, 03:03 PM	#6697
bpothier Member Posts: 20 Karma: 10 Join Date: May 2020 Device: Kobo	Thank you for reply, I was hoping for a quick "Oh, you might have option XX enabled" or something... no need for deep dive I did poke around in Calibre preferences, and under "Saving books to disk" there is an option to "Update metadata in saved copies" option which defaults to enabled (and says "Makes saving to disk slower", so sounds promising). Wondering if this is doing "double updates" where FFF creates an updated ePub which gets imported to Calibre, which then goes back and updates the freshly imported ePub with the Metadata? I'll poke at it a little and will likely update to 5.x at some point and compare...

11-17-2021, 06:50 PM	#6703
bpothier Member Posts: 20 Karma: 10 Join Date: May 2020 Device: Kobo	UPDATE3: TLDR - "fixed" DB performance issue... Library & DB were on NAS, copied metadata.db to local drive and set ENV VAR "CALIBRE_OVERRIDE_DATABASE_PATH" to point to local file.... metadata update of 14 ePUB dropped to 5 seconds. Ref: https://www.mobileread.com/forums/sh...d.php?t=294998 Hopefully this helps someone else with DB perf issues... Details: Spoiler: Upgraded to Calibre 5.32. The 14 books that took 8m to update now took 4m. Still an age, but progress. "Only" around 20k books in Library, DB is around 120MB. Found forum thread - https://www.mobileread.com/forums/sh...d.php?t=294998 - with perf issues around opening/adding/importing. It suggested moving DB file and adding a symlink or setting ENV var. Tried to create windows symlink on NAS share to local disk without success (tried as local admin, changed fsutil R2L symlink option). Setting the ENV variable "CALIBRE_OVERRIDE_DATABASE_PATH=D:\CalibreDB\metad ata.db" does work, though it is less flexible for multiple libraries. After copying DB to local drive, doing FFF metadata update from stored column took... 5 seconds..... Likely will create a few CMD/batch files to set ENV and start Calibre as a workaround. The docs do mention NAS performance issues around locking support, but have not delved into that yet. I don't switch Calibre Libraries very often, so will likely keep it this way for now... Might add a DB copy on startup from local to NAS in the batch script for safety... I think one issue with the DB on the NAS is how Calibre updates the file. I have "recyclebin" enabled on the NAS and when Calibre does a DB update it creates and deletes a temporary metadata.db-journal file. These seem to have accumulated in the NAS recyclebin as "Copy #1234 of metadata.db-journal" and there are now thousands of them...

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] KindleUnpack - The Plugin	DiapDealer	Plugins	523	07-15-2025 06:45 PM
[GUI Plugin] Open With	kiwidude	Plugins	404	02-21-2025 05:42 AM
[GUI Plugin] Marvin XD	Philantrop	Plugins	126	01-29-2017 12:48 PM
[GUI Plugin] KiNotes	-axel-	Plugins	0	07-14-2013 06:39 PM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

Advert

Advert