Aura Manage epub and keep read progress

[geek1011]

Originally Posted by quazardous

@geek1011 share yout tools -> I love command line tools with bash and the joy of gawk lol

- For epub to kepub, I use my open-source tool kepubify (link in my signature).
- For reading ebooks on a computer, I use my web based epub viewer.
- For unpacking and repacking books, I use my tool epubtool (on github)
- For fixing adobe drm tags, I use a regex with sed
- For fixing ocred words, I use some sed expressions
- For fixing misnumbered NCX files, I use a simple awk script, which increments a counter every time it reaches the playorder property
- For fixing bad xml IDs, I use find, xargs, and sed
- I have a bunch of other things too, and I make a lot of one-time utilities and scripts as I need them for more obscure issues with ebooks.

[davidfor]

Originally Posted by quazardous

well it can be user error but I was paying attention to eject and stuff.

The fact is that I've end up with an import loop : so it is not safe by definition. I'm not complaining here I did mess up my kobo ;p

Actually, I'll be a bit stronger than before: It was not the Kobo Utilities plugin that caused this. Something else had to be involved. The Whether it was a bad eject, or a pre-existing corruption, I don't know. Of course, I have no way to prove this other than the very large number of times I have used these functions without breaking anything. Plus the very, very small number of complaints.

I think some of you are over protective with calibre

Protective, maybe, but it is actually fairly insulting to add the word "over" to that. You will probably disagree based on my response. But, you have made a claim about my code, one that you can't really backup (you need to repeat the problem), so maybe I am "over" protective.

lol I do love open source. I m managing web servers. but I also admit that some times things can be messy. And kobo + calibre is messy : lots of wikis are outdated, some options are redundant (metadata plugboard vs driver options). I m sure it will do well at some point but with some efforts and some factory reset.

Everything on the web is messy. Lots of people writing pages that they never maintain. If a random Wiki page out there is out of date, that isn't calibres fault.

As to redundant options, what exactly are you talking about? You mention the metadata plugboards and the driver options. I have to say "Huh?" The only relationship between them is that you can create a plugboard for each combination of driver and the formats the driver supports.

[JSWolf]

Originally Posted by geek1011

I agree about calibre. It is kind of messy. I usually just read the source code, and find out what is done well, what could be done better, and what isn't done at all, then make some standalone tools to do what I want.

My biggest dislikes about calibre is: it's folder structure, how it is hard to version with git, it's meta tags (except for series, and I'm a purist), it's calibre# classes (again, I'm a purist), it's black boxness (I don't know exactly what goes in and out of the db), and it's too many features.

I still use calibre for certain tasks, but then I clean the ebook up afterwards.

For the calibre developers: don't feel bad, I am like this about quite a few other tools. I'm the kind of person who like manually editing ebooks with bash, grep, sed, vim, and vscode. I also still use calibre, a bit in an indirect way.

I've found that with Calibre, if you change some of the settings, you can put books in and get them out without Calibre doing any conversion or metadata update. I can also use the editor without any changes except the ones I make. But if you want this behavior, you have to change some of the default settings in Calibre, the Editor, and the Viewer. But I've done it and I can use Calibre the way I want.

I don't convert ePub to anything unless the code is that sloppy that I want to see what Calibre does to with the mess. I don't convert KF8 to ePub. I use the KindleUnpack to format shift the code and then the editor to clean it up.

I have my ways and when I'm done, the ePub code has not been changed except by me.

[geek1011]

Originally Posted by JSWolf

I've found that with Calibre, if you change some of the settings, you can put books in and get them out without Calibre doing any conversion or metadata update. I can also use the editor without any changes except the ones I make. But if you want this behavior, you have to change some of the default settings in Calibre, the Editor, and the Viewer. But I've done it and I can use Calibre the way I want.

I don't convert ePub to anything unless the code is that sloppy that I want to see what Calibre does to with the mess. I don't convert KF8 to ePub. I use the KindleUnpack to format shift the code and then the editor to clean it up.

I have my ways and when I'm done, the ePub code has not been changed except by me.

The thing is that calibre also insists on it's own directory structure, which I strongly dislike. It is also hard to properly version with things like git, as I never know when something changes.

Also, calibre will always add it's namespace to the opf file when importing a book. It also adds a dc:contributor for itself, a timestamp, and a few more meta properties.

[JSWolf]

Originally Posted by geek1011

The thing is that calibre also insists on it's own directory structure, which I strongly dislike. It is also hard to properly version with things like git, as I never know when something changes.

Also, calibre will always add it's namespace to the opf file when importing a book. It also adds a dc:contributor for itself, a timestamp, and a few more meta properties.

Sorry, but that's wrong. My Calibre setup does not do any of those changes to the ePub when added to the library. LIie I said before, you just have to set the correct options to get Calibre to not modify the ePub automatically.

In the Editor, Edit > Preferences > Integration with calibre
Uncheck "Update metadata embedded in the book when opening"

In the Viewer, Preferences > Miscellaneous options
Uncheck "Keep a copy of bookmarks/current page inside the e-book file, for easy sharing"

I'm hoping this is correct. In Calibre, Preferences, Adding books > Adding actions
Uncheck "Automatically convert added books to the current output format" and "When auto-converting, convert even if the format being added is the same as the output format"

Next in Calibre, Preferences > Saving books to disk
Uncheck "Save cover separately" and "Update metadata in saved copies"

That might be all you need to to stop the ePub from being automatically being changed when you don't want it to.

[geek1011]

Originally Posted by JSWolf

Sorry, but that's wrong. My Calibre setup does not do any of those changes to the ePub when added to the library. LIie I said before, you just have to set the correct options to get Calibre to not modify the ePub automatically.

In the Editor, Edit > Preferences > Integration with calibre
Uncheck "Update metadata embedded in the book when opening"

In the Viewer, Preferences > Miscellaneous options
Uncheck "Keep a copy of bookmarks/current page inside the e-book file, for easy sharing"

I'm hoping this is correct. In Calibre, Preferences, Adding books > Adding actions
Uncheck "Automatically convert added books to the current output format" and "When auto-converting, convert even if the format being added is the same as the output format"

Next in Calibre, Preferences > Saving books to disk
Uncheck "Save cover separately" and "Update metadata in saved copies"

That might be all you need to to stop the ePub from being automatically being changed when you don't want it to.

OK, I didn't know about that first option, or the last one you mentioned. That does everything except for the fact that then, there is not really any point of using calibre.

Also, that still does not let me use my own directory structure.

[JSWolf]

Originally Posted by geek1011

OK, I didn't know about that first option, or the last one you mentioned. That does everything except for the fact that then, there is not really any point of using calibre.

Also, that still does not let me use my own directory structure.

I don't mind the directory structure. It's all behind the scenes. So I don't have to deal with it. I just use the GUI front end and when I want to pull eBooks from Calibre, it's not an issue.

So why is not having Calibre automatically modify the eBook on import, opening with the Editor or Viewer meaning that there is not really any point to using Calibre? I use Calibre a lot since I figured out how to stop it from modifying when I don't want it to.

[davidfor]

Originally Posted by JSWolf

I don't mind the directory structure. It's all behind the scenes. So I don't have to deal with it. I just use the GUI front end and when I want to pull eBooks from Calibre, it's not an issue.

So why is not having Calibre automatically modify the eBook on import, opening with the Editor or Viewer meaning that there is not really any point to using Calibre? I use Calibre a lot since I figured out how to stop it from modifying when I don't want it to.

I'm not sure if I see the point of importing the book, maintaining the metadata in calibre but not writing that metadata back to the book at sometime. Because if I don't then, when I look at the book on my devices, then I won't have the right title, author, description or cover.

And don't say to use the Modify ePub plugin, the Embed metadata function or a conversion (yes, I know you don't) to do this. Then you are back to the situation that geek1011 is complaining about. All of these methods update the metadata in the book in exactly the same way. They will all put the things in the OPF that geek1011 doesn't like.

@geek1011: I'm not a fan of how the extra metadata is written to the OPF, and every now and then I consider looking at how to block it. Adding an option to only write the DC metadata, or calibre core metadata is probably not hard. But, it just hasn't been important. I have almost zero need to look at the OPF and I know that while there is a lot of seeming cruft, it is all valid and usable if the application wants it.

As to the library file structure, it's a database. I treat it that way and it doesn't bug me. I do play in it, but I do understand what is happening and knowing when the files will be updated is easy: when I change something. If I change some metadata, the OPF file will change. And the book will change if I push the change to the book. And if I change some common metadata element (rename a tag, fix an authors name) then all the OPF files for books with that element will change. Calibre updates the database and then updates the files in the background, so this can take a bit of time for a change that affects a lot of books. But if you want to put the library into a version control system, then you really have to just accept all changes. Push them once a day or each time you make bulk changes or something like that.

[geek1011]

Originally Posted by davidfor

I'm not sure if I see the point of importing the book, maintaining the metadata in calibre but not writing that metadata back to the book at sometime. Because if I don't then, when I look at the book on my devices, then I won't have the right title, author, description or cover.

And don't say to use the Modify ePub plugin, the Embed metadata function or a conversion (yes, I know you don't) to do this. Then you are back to the situation that geek1011 is complaining about. All of these methods update the metadata in the book in exactly the same way. They will all put the things in the OPF that geek1011 doesn't like.

@geek1011: I'm not a fan of how the extra metadata is written to the OPF, and every now and then I consider looking at how to block it. Adding an option to only write the DC metadata, or calibre core metadata is probably not hard. But, it just hasn't been important. I have almost zero need to look at the OPF and I know that while there is a lot of seeming cruft, it is all valid and usable if the application wants it.

As to the library file structure, it's a database. I treat it that way and it doesn't bug me. I do play in it, but I do understand what is happening and knowing when the files will be updated is easy: when I change something. If I change some metadata, the OPF file will change. And the book will change if I push the change to the book. And if I change some common metadata element (rename a tag, fix an authors name) then all the OPF files for books with that element will change. Calibre updates the database and then updates the files in the background, so this can take a bit of time for a change that affects a lot of books. But if you want to put the library into a version control system, then you really have to just accept all changes. Push them once a day or each time you make bulk changes or something like that.

Yeah. I'm kind of overly pure about ebook formats. I usually remove everything but cover (I scale it, and convert it to JPG), title, author (I fix the sorting to match the epub spec), series, publisher, description (with custom fonts and styles stripped), date, and a few more elements. I also usually rename CSS classes, and strip extraneous styles. I really cannot stand seeing a cluttered epub file. I'd use fb2 if it had some more formatting features, better compatibility, and worked better with nonfiction. It usually takes me about 4-6 minutes to go through each new book.

As for VCS, I like to track changes properly. I usually store my epubs unzipped, with CI/CD setup to zip them and push them to my devices. Also, using a sqlite database in VCS gets large quite quickly.

To update metadata, I prefer to do it manually in the command line. I find I have more control that way, and it is more flexible.

Also, don't get me wrong: Calibre is great for most people, it's just that I kind of have the opposite ideology. I'm still grateful for the devs, as I learn quite a bit from reading the calibre source code (yes, I converted it into a book ). IMO it's the best documentation for some obsure ebook formats.

[JSWolf]

Originally Posted by davidfor

I'm not sure if I see the point of importing the book, maintaining the metadata in calibre but not writing that metadata back to the book at sometime. Because if I don't then, when I look at the book on my devices, then I won't have the right title, author, description or cover.

And don't say to use the Modify ePub plugin, the Embed metadata function or a conversion (yes, I know you don't) to do this. Then you are back to the situation that geek1011 is complaining about. All of these methods update the metadata in the book in exactly the same way. They will all put the things in the OPF that geek1011 doesn't like.

If I want to change the metadata, I use the Calibre editor and edit the OPF myself. I have seen enough eBooks where the title was just plain wrong.

[davidfor]

Originally Posted by JSWolf

If I want to change the metadata, I use the Calibre editor and edit the OPF myself. I have seen enough eBooks where the title was just plain wrong.

Really? You don't fix the title in the library, just edit the book and manually fix it in the OPF?

Yes, Jon, that is what it sounds like you are saying you are doing. And the idea of manually editing the OPF when you don't need to is, to me, stupid. The risk of making a mistake is to high to make it worthwhile. Calibre updates this perfectly well including making sure there are no illegal character.

What geek1011 is doing is very different, and to me over the top Scripting the changes or developing personal tools to do this does make sense, but I'm far to lazy not to take advantage of the large code base to make some things easy for me. I do a lot of what geek1011 does (clean the code and styles), but I have my own plugins to do some of the grunt work and am happy with the calibre editor for the rest. With a lot of saved searches. But manually maintaining the metadata in the OPF while using the editor, that's a complete waste of time.

[geek1011]

Originally Posted by davidfor

Really? You don't fix the title in the library, just edit the book and manually fix it in the OPF?

Yes, Jon, that is what it sounds like you are saying you are doing. And the idea of manually editing the OPF when you don't need to is, to me, stupid. The risk of making a mistake is to high to make it worthwhile. Calibre updates this perfectly well including making sure there are no illegal character.

What geek1011 is doing is very different, and to me over the top Scripting the changes or developing personal tools to do this does make sense, but I'm far to lazy not to take advantage of the large code base to make some things easy for me. I do a lot of what geek1011 does (clean the code and styles), but I have my own plugins to do some of the grunt work and am happy with the calibre editor for the rest. With a lot of saved searches. But manually maintaining the metadata in the OPF while using the editor, that's a complete waste of time.

TBH, I am faster when editing the opf manually. With an XML checker integrated in my IDE, epubcheck on save, and repacking Epub on close, it's actually quite efficient.

Also, as for it being OTT, I live in the command line, so it's not anything too different than what I usually do. To me, calibre is over the top

Btw, https://ux.stackexchange.com/questio...les-still-used is quite an interesting read.

[davidfor]

Originally Posted by geek1011

TBH, I am faster when editing the opf manually. With an XML checker integrated in my IDE, epubcheck on save, and repacking Epub on close, it's actually quite efficient.

Also, as for it being OTT, I live in the command line, so it's not anything too different than what I usually do. To me, calibre is over the top

I live on the command line at work (I'm swearing at tcpdump and syslog today). But use GUI based editors as much as possible.

Btw, https://ux.stackexchange.com/questio...les-still-used is quite an interesting read.

I'll have a look later.

[geek1011]

Originally Posted by davidfor

I live on the command line at work (I'm swearing at tcpdump and syslog today). But use GUI based editors as much as possible.

I use vim and VSCode equally. But some things are better from the command line, and as for this book stuff, I use VSCode for the more complex tasks, as it is more featured than Calibre for the types of modifications I make to ebooks. Also, Calibre is a bit tricky to navigate with only the keyboard.