Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 05-30-2011, 11:51 PM   #61
capnm
Groupie
capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'
 
Posts: 150
Karma: 10001
Join Date: Feb 2011
Device: sony
Yessss .... this has evolved hasn't it

Is this where you're going?

Search & remove tools for three groups of junk (to some) files:
calibre_bookmarks.txt (which some people will want to keep)
itunes plist/artwork (which I suppose somebody will want to keep )
thumbs.db/.DS_store (OS artifacts nobody should want)

I think I missed some conversation, but are you going to search for these whether or not they're in the manifest and de-manifest them where needed if found?

Then, handled a little differently, search for & log & possibly remove unmanifested files (other than the cases above)?

That sounds really good to me.

I'd treat the itunes artwork just like the plist files.
capnm is offline   Reply With Quote
Old 05-31-2011, 03:47 AM   #62
Agama
Guru
Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.
 
Agama's Avatar
 
Posts: 667
Karma: 436517
Join Date: Jul 2010
Location: UK
Device: PRS-300 (R.I.P.), PW2, Nexus7
Quote:
Originally Posted by kiwidude View Post
Changes in this release:
  • Add option to remove non dc: metadata elements
Thanks! That works exactly as required.

I have a few queries re the "Add unmanifested files to manifest" option:

1) It works but it changes the media-type from application/xhtml+xml to text/html - Is this OK? (The 'text' files in ePubs are xhtml rather than html).

2) The added items all appear on a single line within the <manifest> section. Would it be possible to add each as a separate line? I know it's only cosmetic but it would be nice!

3) Should this option and "Remove unmanifested files from ePub" be mutally exclusive?

Last edited by Agama; 05-31-2011 at 04:00 AM. Reason: Add query 3
Agama is offline   Reply With Quote
 
Advertisement
Old 05-31-2011, 05:26 AM   #63
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by capnm View Post
Yessss .... this has evolved hasn't it

Is this where you're going?

Search & remove tools for three groups of junk (to some) files:
calibre_bookmarks.txt (which some people will want to keep)
itunes plist/artwork (which I suppose somebody will want to keep )
thumbs.db/.DS_store (OS artifacts nobody should want)

I think I missed some conversation, but are you going to search for these whether or not they're in the manifest and de-manifest them where needed if found?

Then, handled a little differently, search for & log & possibly remove unmanifested files (other than the cases above)?

That sounds really good to me.

I'd treat the itunes artwork just like the plist files.
With my last post I hadn't considered that people would want to keep plist files but yes I guess maybe some will if they continue to view via iTunes. So what you list above would be the three options of bookmarks / iTunes / OS artifacts.

And yes the change I made in 0.2.1 was to remove such files from the manifest if they had been accidentally put in there.

Quality check does the "search" and logging, that will not change. But yes the remove unmanifested and add unmanifested files options will handle the "everything else" files.
kiwidude is offline   Reply With Quote
Old 05-31-2011, 05:33 AM   #64
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by Agama View Post
Thanks! That works exactly as required.

I have a few queries re the "Add unmanifested files to manifest" option:

1) It works but it changes the media-type from application/xhtml+xml to text/html - Is this OK? (The 'text' files in ePubs are xhtml rather than html).
it should be the same logic that Kovid had in his ePub-fix code which tries to guess the type from the extension. I will take a look.
Quote:
Originally Posted by Agama View Post
2) The added items all appear on a single line within the <manifest> section. Would it be possible to add each as a separate line? I know it's only cosmetic but it would be nice!
the elements need a tail added to them. Perhaps Kovid can be convinced to change his function in the container class to support this. Otherwise I will add yet another override to my derived class.
Quote:
Originally Posted by Agama View Post
3) Should this option and "Remove unmanifested files from ePub" be mutally exclusive?
They are if you look at the tooltip and the code.
kiwidude is offline   Reply With Quote
Old 05-31-2011, 08:17 AM   #65
Agama
Guru
Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.
 
Agama's Avatar
 
Posts: 667
Karma: 436517
Join Date: Jul 2010
Location: UK
Device: PRS-300 (R.I.P.), PW2, Nexus7
Quote:
1) It works but it changes the media-type from application/xhtml+xml to text/html - Is this OK? (The 'text' files in ePubs are xhtml rather than html).

it should be the same logic that Kovid had in his ePub-fix code which tries to guess the type from the extension. I will take a look.
This is probably my fault then! I have used the .htm extension because it's easy to preview the file in a browser, (proof-reading), but have then used:

<?xml version='1.0' encoding='utf-8'?>
<html xmlns="http://www.w3.org/1999/xhtml">

in the files to declare them as xhtml for ePub purposes, (with media-type="application/xhtml+xml" in the manifest.)
Agama is offline   Reply With Quote
Old 05-31-2011, 08:58 AM   #66
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
2 remarks:
Your trash-cleaner sounds nice. I think you could make this an option in a settings-element.
I want to remove thumbs.db, iTunesArtwork, .DS_Store but for example, I would also like to remove ~... .~ .temp .tmp ..... a lot more files.
So, first I lookup books with this kind of files, after that, I look at the usage of these files. Until now, I did not see any useful files of this kind, so I removed them.
Same as with some quality check files I found using your unmanifested plugin. It seems a tool on my pc runs quality-checks and adds a folder with results. This file can be found using the unmanifested files. But I also found a book where this file was unmanifested, but another file in the same folder (also a txt (or xml don't know any more) file with check info) was manifested.
I can not find this file with your tool because it seems a 'legal' file. If I can add custom filetypes (complete filenames) to find with quality-check, it would be nice to have the option to remove that kind of files (cross-config usage? (get out of the sand )). And maybe even the empty folder if it is not a default epub-folder and there are no other references any more (like /testresults).
p.s. remember to ad a good tooltip with explicite mention of removed files (thumbs.db, plist and .DS_Store) and remember you will have to update this with every addition of files to this function.

The second remark: I ran a batch of remove-task on some files today.
In most cases it is useful to leave the temp files. In your tool not I think. So would it be possible to let this plugin remove its temp that have been updated / canceld?
I now have 12 folders, 1424 files and 857.605.545 bytes in my cache-file that will
1. be removed when calibre closes (slow down calibre)
2. stay when calibre crashes
3. consume space during calibre run while
4. never be used by calibre or the epub-plugin

But still want to say again a great product
drMerry is offline   Reply With Quote
Old 05-31-2011, 12:02 PM   #67
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,300
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by kiwidude View Post
the elements need a tail added to them. Perhaps Kovid can be convinced to change his function in the container class to support this. Otherwise I will add yet another override to my derived class.
Go for it. I have no objections to adding that to the container class.
kovidgoyal is offline   Reply With Quote
Old 05-31-2011, 03:51 PM   #68
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by Agama View Post
This is probably my fault then! I have used the .htm extension because it's easy to preview the file in a browser, (proof-reading), but have then used:

<?xml version='1.0' encoding='utf-8'?>
<html xmlns="http://www.w3.org/1999/xhtml">

in the files to declare them as xhtml for ePub purposes, (with media-type="application/xhtml+xml" in the manifest.)
Hmmm. Ok, so that looks like a limitation of that approach the epub-fix code uses of just using file extensions. I looked at a few random epubs and found they were similar of .html files that actually contain xhtml.

I think I need to change the behaviour (in my plugin anyways) to cater for this. I guess the question is what "defines" a file as being xhtml? Should I look for the xmlns declaration or just the xml declaration?
kiwidude is offline   Reply With Quote
Old 05-31-2011, 04:35 PM   #69
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,300
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
xmlns is the canonical test
kovidgoyal is offline   Reply With Quote
Old 05-31-2011, 05:04 PM   #70
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by kovidgoyal View Post
xmlns is the canonical test
Thx Kovid,

@Agama - this will be sorted for the next plugin version. I will have to wait for the next Calibre release to publish it here though, as I had to make a change to Calibre for this and the "prettying" of the xml inserted which was the other issue you raised.
kiwidude is offline   Reply With Quote
Old 06-03-2011, 04:36 PM   #71
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
v0.2.2 Beta

Changes in this release:
  • Treat iTunesArtwork the same as iTunes plist files
  • Add an option to remove OS artifacts of .DS_Store and thumbs.db files
  • Ensure that any xml elements inserted in the manifest are "tailed" correctly for indenting
  • When adding items to manifest, if a .htm* file check for xmlns indicating mimetype of xhtml+xml

This requires Calibre 0.8.4 and ensure most of the additions/suggestions raised above are included.

The major thing missing from this plugin now from my list is supporting cover image replacement. That won't be trivial to add so I'm not rushing at this point

Last edited by kiwidude; 06-06-2011 at 01:13 PM. Reason: Removed attachment as later version in thread
kiwidude is offline   Reply With Quote
Old 06-05-2011, 11:46 AM   #72
Agama
Guru
Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.Agama ought to be getting tired of karma fortunes by now.
 
Agama's Avatar
 
Posts: 667
Karma: 436517
Join Date: Jul 2010
Location: UK
Device: PRS-300 (R.I.P.), PW2, Nexus7
Quote:
Originally Posted by kiwidude View Post
Changes in this release:
  • When adding items to manifest, if a .htm* file check for xmlns indicating mimetype of xhtml+xml
Thanks. This works nicely.
Agama is offline   Reply With Quote
Old 06-05-2011, 04:20 PM   #73
capnm
Groupie
capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'
 
Posts: 150
Karma: 10001
Join Date: Feb 2011
Device: sony
A slight hiccup while trying to remove calibre_bookmarks.
Several files failed with the following:

ERROR: Traceback (most recent call last):
File "calibre_plugins.modify_epub.modify", line 81, in process_book
File "calibre_plugins.modify_epub.container", line 109, in write
File "site-packages\calibre\utils\zipfile.py", line 1208, in write
File "site-packages\calibre\utils\zipfile.py", line 370, in FileHeader
error: ushort format requires 0 <= number <= USHRT_MAX

Oddly, they're epubs where all the stored files are all lacking timestamps

I have a sinking feeling this is related to your patch to preserve the timestamps (and some peculiar epub makers).


More:
tweak epub also chokes on these:
calibre, version 0.8.4
ERROR: Unhandled exception: <b>error</b>:ushort format requires 0 <= number <= USHRT_MAX

Traceback (most recent call last):
File "site-packages\calibre\gui2\dialogs\tweak_epub.py", line 108, in preview
File "site-packages\calibre\gui2\dialogs\tweak_epub.py", line 86, in do_rebuild
File "zipfile.py", line 1071, in write
File "zipfile.py", line 348, in FileHeader
error: ushort format requires 0 <= number <= USHRT_MAX

Last edited by capnm; 06-05-2011 at 04:25 PM. Reason: more info
capnm is offline   Reply With Quote
Old 06-05-2011, 04:48 PM   #74
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@capnm - Funnily enough I have seen that exact same error once since that timestamp change was made too. Unfortunately in my process of investigating exactly the problem I somehow caused the error to not happen again and I havent found another file to replicate it with since.

As you point out it happens with Tweak ePub so it is nothing to do with the plugin and I believe everything to do with that change we made to set timestamps on the unzipped files. From memory it doesn't happen at the unzip/explode stage (where the change to the code was made) but instead happens when the zip file is attempted to be rebuilt.

I *definitely* would not want the patch reversed out as have found it way too useful, but obviously we have to fix this problem with it. Any chance you could attach the ePub to a ticket for someone to take a look at?
kiwidude is offline   Reply With Quote
Old 06-05-2011, 04:58 PM   #75
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,300
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I've added code to set the timestamp to 1980-1-1 if it overflows, which will prevent that error, but as to why it is happening in the first place, I haven't a clue.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Any web-to-epub plugin for internet browser? bthoven ePub 7 07-10-2011 06:14 AM
[Old Thread] Reading epub on viewer inexplicably changes the time stamp of epub greenapple Library Management 20 03-19-2011 11:18 PM
Easy way to modify thread subscription emails in bulk? snipenekkid Feedback 11 02-06-2011 04:47 AM
Another plugin dev question DiapDealer Plugins 2 12-11-2010 02:46 PM
Epub plugin dev DiapDealer Plugins 15 11-12-2010 10:36 AM


All times are GMT -4. The time now is 02:15 PM.


MobileRead.com is a privately owned, operated and funded community.