[Plugin] RemoveUnusedBookmarks - removes all unused bookmarks from epubs - Page 2

DiapDealer · 03-26-2018, 09:36 PM

No problem.

The mistake you're making is think of them (meta tag name attributes) as "ids" in the first place. They're not. They're not potential targets for hrefs at all. They serve a different purpose.

slowsmile · 03-26-2018, 11:40 PM

Update: The following changes have been made(v0.1.2):

* The plugin now removes the meta line if it contains the 'name="calibre:cover"' attribute in the cover file.

* The plugin will now not change any "name" attribute values or names in any meta tags.

Maui · 03-27-2018, 04:16 AM

Another approach would be to not start "souping" on top but on <body>

body = soup.body
tags = body.find_all(True)

KevinH · 03-27-2018, 10:47 AM

Also be careful of page tables in either epub3's nav xhtml and for page tables in the ncx or adobe page tables that use those ids for navigation.

slowsmile · 03-27-2018, 09:50 PM

Update: Fixed a problem that was causing svg formatting errors in Epubcheck. SVG images can now be used in epubs without problems when using this plugin.(v0.1.3)

slowsmile · 03-28-2018, 06:08 AM

@KevinH...Regarding your epub 3 nav xhtml advice - this plugin only supports epub 2 files. It also only checks for and removes ids(aka bookmarks in plain speak) that are not associated with any internal doc links, which limits these changes to just the xhtml files in the epub -- it also doesn't remove or change anything in the ncx file. So, hopefully, there shouldn't be any of the problems that you've previously mentioned occurring with this plugin.

DiapDealer · 03-28-2018, 06:20 AM

Quote:

Originally Posted by slowsmile

@KevinH...Regarding your epub 3 nav xhtml advice -- this plugin will only work with epub 2 files. It also only checks and removes ids(aka bookmarks in plain speak) that are not associated with any internal doc links, which limits these changes to just the xhtml files in the epub -- it doesn't remove or change anything in the ncx file. So, hopefully, there shouldn't be any of the problems that you've previously mentioned with this plugin.

But since the ncx can contain links to ids in the xhtml, how can you remove ids as "unused" if you don't parse the ncx to check if it's using them?

slowsmile · 03-28-2018, 07:58 AM

@DiapDealer...While I was writing the plugin code I thought about what you you've said above and realized that all the internal links in the xhtml file are a mixture of either page links or links to ids(boookmarks). The ncx and opf files also contain links -- but these are mostly page links as a rule -- not links to an id or bookmark. And any href + id in the opf and ncx will already be in the xhtml file anyway - ie there are no ids(bookmarks) in the opf or ncx that are referenced by links in the opf or ncx.

The way the plugin works is that it initially gathers all hrefs with ids and splits these and saves each href id into a list. On the second pass through the xhtml files every id is compared to all the ids in the href id list. If the id is not there in the href list it's removed, if the id is there then the id is left unchanged. And because of the way I've done it, no internal links are ever changed or removed because all the plugin code is doing is just removing ids that are not used in only the xhtml files. And if these removed ids are not used in the xhtml then they won't be used in the links in either the opf or ncx.

KevinH · 03-28-2018, 08:35 AM

That is not true in general. You can have a link in the ncx that uses a fragment that points to an "id" in an xhtml file that may not be used anyplace else in the xhtml.

To be safe, simply walk the ncx, collecting any links with fragments and add that fragment info to your used id list. You should probably do the same for the guide section of the opf to be 100% safe as well but fragment use in guide items is very rare.

DiapDealer · 03-28-2018, 08:52 AM

Quote:

Originally Posted by slowsmile

The ncx and opf files also contain links -- but these are mostly page links as a rule -- not links to an id or bookmark. And any href + id in the opf and ncx will already be in the xhtml file anyway - ie there are no ids(bookmarks) in the opf or ncx that are referenced by links in the opf or ncx.

This is faulty logic that will allow many exceptions to fall through the cracks. Your plugin has the potential to break ncx navigation. You should take the (known) steps to prevent it.

slowsmile · 03-28-2018, 08:56 AM

@Kevin...Sorry but I disagree with you. The best way to understand it is to ask the question: Where does the opf and ncx get these hrefs with ids from?

The only place that the opf or ncx can get hrefs with ids is from the epub toc file which is an xhtml file whose ids have already been checked by the plugin. The ebook image links and guide links in the opf will all be page links anyway(without ids). And even if the guides have hrefs with ids these will also come from the toc file via the opf. I've also made sure that any ids that contain "toc", "cover", "text" or "start"(for iBook epubs only) are not removed from the xhtml files because I think there are still ereaders out there that use these ids rather than the guide information to generate the Go To menu links on the device.

BeckyEbook · 03-28-2018, 09:27 AM

@slowsmile. Look at the attachment. This is a free epub file ideal for testing your plugins.

slowsmile · 03-28-2018, 09:40 AM

@Becky...Thanks very much for that epub - I'll definitely use it for testing. Well, Quo Vadis - I remember seeing that story as a film when it first came out all those years ago.

DiapDealer · 03-28-2018, 09:54 AM

You're not getting it.

NCX entry:

Code:

<navPoint id="navpoint-2" playOrder="2">
      <navLabel>
        <text>Chapter  Four</text>
      </navLabel>
      <content src="Text/part_one.xhtml#chap004"/>
    </navPoint>

the xhtml (a portion of part_one.xhtml):

Code:

<h3 id="chap001">Chapter One</h3>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>

<h3 id="chap002">Chapter Two</h3>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>

<h3 id="chap003">Chapter Three</h3>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>

<h3 id="chap004">Chapter Four</h2>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>
<p> blah, blah, blah</p>

This particular epub (like many) has no html toc page. Only an ncx. You're completely wrong that an html toc is the only place that an ncx can "get hrefs with ids". You're making the mistake of only basing your reasoning on epubs that meet your particular criteria. Epubs don't have to have an html toc, so you need to allow for the fact that users will be using your plugin on such epubs

Your plugin will remove the id from <h3 id="chap004"> (because your parsing of all of the epub's xhtml reveals no hrefs that utilize the "chap004" id/fragment ) which will break the functionality of a user selecting Chapter Four from their reader's ncx-based navigation utility. Instead of going to the Chapter Four portion of the part_one.xhtml file, it will default to the beginning of the file (Chapter One), because the target id has been deleted for the entry they want.

slowsmile · 03-28-2018, 10:12 AM

@Becky...I ran Epubcheck on your Quo Vadis test epub and had many errors caused by bad ids in the <body> tag and elsewhere. So I ran the IDErrorCheck plugin on your file first which successfully cleared all the bad ids. Then I ran the RemoveUnusedBookmarks plugin and that removed 241 unused ids. Finally, I ran Epubcheck to see if all errors were gone and it passed without error.

That test file was a great test for both plugins. Thanks again.

03-27-2018, 04:16 AM	#18
Maui Connoisseur Posts: 57 Karma: 600000 Join Date: Jan 2018 Device: Galaxy Tab S2	Another approach would be to not start "souping" on top but on <body> body = soup.body tags = body.find_all(True) Last edited by Maui; 03-27-2018 at 04:23 AM.

03-28-2018, 06:08 AM	#21
slowsmile Witchman Posts: 628 Karma: 788808 Join Date: May 2013 Location: Philippines Device: Android S5	@KevinH...Regarding your epub 3 nav xhtml advice - this plugin only supports epub 2 files. It also only checks for and removes ids(aka bookmarks in plain speak) that are not associated with any internal doc links, which limits these changes to just the xhtml files in the epub -- it also doesn't remove or change anything in the ncx file. So, hopefully, there shouldn't be any of the problems that you've previously mentioned occurring with this plugin. Last edited by slowsmile; 03-28-2018 at 06:35 AM.

03-28-2018, 07:58 AM	#23
slowsmile Witchman Posts: 628 Karma: 788808 Join Date: May 2013 Location: Philippines Device: Android S5	@DiapDealer...While I was writing the plugin code I thought about what you you've said above and realized that all the internal links in the xhtml file are a mixture of either page links or links to ids(boookmarks). The ncx and opf files also contain links -- but these are mostly page links as a rule -- not links to an id or bookmark. And any href + id in the opf and ncx will already be in the xhtml file anyway - ie there are no ids(bookmarks) in the opf or ncx that are referenced by links in the opf or ncx. The way the plugin works is that it initially gathers all hrefs with ids and splits these and saves each href id into a list. On the second pass through the xhtml files every id is compared to all the ids in the href id list. If the id is not there in the href list it's removed, if the id is there then the id is left unchanged. And because of the way I've done it, no internal links are ever changed or removed because all the plugin code is doing is just removing ids that are not used in only the xhtml files. And if these removed ids are not used in the xhtml then they won't be used in the links in either the opf or ncx. Last edited by slowsmile; 03-28-2018 at 08:30 AM.

03-28-2018, 08:56 AM	#26
slowsmile Witchman Posts: 628 Karma: 788808 Join Date: May 2013 Location: Philippines Device: Android S5	@Kevin...Sorry but I disagree with you. The best way to understand it is to ask the question: Where does the opf and ncx get these hrefs with ids from? The only place that the opf or ncx can get hrefs with ids is from the epub toc file which is an xhtml file whose ids have already been checked by the plugin. The ebook image links and guide links in the opf will all be page links anyway(without ids). And even if the guides have hrefs with ids these will also come from the toc file via the opf. I've also made sure that any ids that contain "toc", "cover", "text" or "start"(for iBook epubs only) are not removed from the xhtml files because I think there are still ereaders out there that use these ids rather than the guide information to generate the Go To menu links on the device. Last edited by slowsmile; 03-28-2018 at 09:48 AM.

03-28-2018, 09:54 AM	#29
DiapDealer Grand Sorcerer Posts: 27,546 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	You're not getting it. NCX entry: Code: <navPoint id="navpoint-2" playOrder="2"> <navLabel> <text>Chapter Four</text> </navLabel> <content src="Text/part_one.xhtml#chap004"/> </navPoint> the xhtml (a portion of part_one.xhtml): Code: <h3 id="chap001">Chapter One</h3> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <h3 id="chap002">Chapter Two</h3> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <h3 id="chap003">Chapter Three</h3> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <h3 id="chap004">Chapter Four</h2> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <p> blah, blah, blah</p> <p> blah, blah, blah</p> This particular epub (like many) has no html toc page. Only an ncx. You're completely wrong that an html toc is the only place that an ncx can "get hrefs with ids". You're making the mistake of only basing your reasoning on epubs that meet your particular criteria. Epubs don't have to have an html toc, so you need to allow for the fact that users will be using your plugin on such epubs Your plugin will remove the id from <h3 id="chap004"> (because your parsing of all of the epub's xhtml reveals no hrefs that utilize the "chap004" id/fragment ) which will break the functionality of a user selecting Chapter Four from their reader's ncx-based navigation utility. Instead of going to the Chapter Four portion of the part_one.xhtml file, it will default to the beginning of the file (Chapter One), because the target id has been deleted for the entry they want.

03-26-2018, 09:36 PM	#16
DiapDealer Grand Sorcerer Posts: 27,546 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	No problem. The mistake you're making is think of them (meta tag name attributes) as "ids" in the first place. They're not. They're not potential targets for hrefs at all. They serve a different purpose.

03-26-2018, 11:40 PM	#17
slowsmile Witchman Posts: 628 Karma: 788808 Join Date: May 2013 Location: Philippines Device: Android S5	Update: The following changes have been made(v0.1.2): * The plugin now removes the meta line if it contains the 'name="calibre:cover"' attribute in the cover file. * The plugin will now not change any "name" attribute values or names in any meta tags.

03-27-2018, 10:47 AM	#19
KevinH Sigil Developer Posts: 7,630 Karma: 5433388 Join Date: Nov 2009 Device: many	Also be careful of page tables in either epub3's nav xhtml and for page tables in the ncx or adobe page tables that use those ids for navigation.

03-27-2018, 09:50 PM	#20
slowsmile Witchman Posts: 628 Karma: 788808 Join Date: May 2013 Location: Philippines Device: Android S5	Update: Fixed a problem that was causing svg formatting errors in Epubcheck. SVG images can now be used in epubs without problems when using this plugin.(v0.1.3)

03-28-2018, 08:35 AM	#24
KevinH Sigil Developer Posts: 7,630 Karma: 5433388 Join Date: Nov 2009 Device: many	That is not true in general. You can have a link in the ncx that uses a fragment that points to an "id" in an xhtml file that may not be used anyplace else in the xhtml. To be safe, simply walk the ncx, collecting any links with fragments and add that fragment info to your used id list. You should probably do the same for the guide section of the opf to be 100% safe as well but fragment use in guide items is very rare.

03-28-2018, 09:40 AM	#28
slowsmile Witchman Posts: 628 Karma: 788808 Join Date: May 2013 Location: Philippines Device: Android S5	@Becky...Thanks very much for that epub - I'll definitely use it for testing. Well, Quo Vadis - I remember seeing that story as a film when it first came out all those years ago.

03-28-2018, 10:12 AM	#30
slowsmile Witchman Posts: 628 Karma: 788808 Join Date: May 2013 Location: Philippines Device: Android S5	@Becky...I ran Epubcheck on your Quo Vadis test epub and had many errors caused by bad ids in the <body> tag and elsewhere. So I ran the IDErrorCheck plugin on your file first which successfully cleared all the bad ids. Then I ran the RemoveUnusedBookmarks plugin and that removed 241 unused ids. Finally, I ran Epubcheck to see if all errors were gone and it passed without error. That test file was a great test for both plugins. Thanks again.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Removing plugin in Calibre removes it from all instances of Calibre	oblox	Calibre	9	09-09-2016 05:39 AM
iPad Possibility to sync bookmarks through side loaded ePubs (Any iOS software?)	andsoitgoes	Apple Devices	12	04-13-2012 07:38 PM
Modify bookmarks in epubs	silentguy	Development	3	08-03-2011 05:37 PM
Sideloaded ePubs, chapters and bookmarks	Steven Lyle Jordan	Nook Color & Nook Tablet	10	02-05-2011 06:35 PM
Problem (bug) with bookmarks in PDF plugin (2.0 RC2)	luite	iRex	1	07-12-2010 02:36 AM