Kindle Mac app, automatic adding

[emmgee]

I finally sat down to learn how to use Calibre. I have 300+ books in my Amazon Kindle "library" and want to store them in Calibre so that I can organize them via metadata and also keep track of which books I've read. Calibre is quite complex...

I decided to install the OS X Kindle App because I felt it would be easier to download books then copy them into Calibre without having to connect a device to my laptop. I discovered how to add books automatically when they are downloaded to the Kindle storage folder. So far so good.

I downloaded and categorized about 100 books and then closed up shop. When I launched the Kindle app, I saw that there were no downloaded books! The files are in the directory but for some reason the Kindle app doesn't think they have been downloaded. I played with this and can reproduce the problem 100% of the time.

If I use the "Add books from a single directory" function, this doesn't happen. But with that function I either have to know exactly which cryptically named file is the most recently downloaded book (e.g., B005G05WDO_EBOK.aza) or I have to select everything in the directory and then contend with the duplicate book dialog.

Is there a way to do this automatically?

[kovidgoyal]

Automatic adding deletes added files. If you want to know what the most recently downloaded file is, simply sort the finder window by timestamp instead of filename.

[emmgee]

LOL – "Simply"... In order to point the "Add books" feature to the correct folder on the Mac, the user has to figure this out first. That is not simple. By default it is a folder within the user library folder, something that Apple has make it difficult to access.

To make the folder easier to point to in Calibre trick is to change default folder in the Kindle app. That's what I did. But the user has to first imagine this is possible and then explore how to do it in the app's preferences panel. Once this is done, access to the folder is via the "Add books" feature is simple.

But the cryptic file naming convention makes things not so simple. If the number of new books is greater than 6 (that's how many azw files are displayed in the "Add books from a single directory" dialog), then it is really hard to figure out which file is new and which isn't. No, I don't think this approach is simple at all when the entire workflow is considered.

The best approach that I found is to use the third option in the "Add books" dropdown. You click on that (add multiple books from the same directory), click on "Open", wait for the read metadata/add books process to complete, then select "Select None". This way, duplicate books are not added – if you are lucky. Each time I have done this a couple of previously added books still show up and I have to delete them.

I think this is a very common use case: "Download new books from Mac (or Windows) Kindle app, add newly added books to Calibre". I would think this should a one-button operation within Calibre. That would be simple.

[kovidgoyal]

Well you are using two instances of very poorly designed software -- the Kindle app and Finder. It cannot be calibre's job to compensate for the bad design of random external software.

If you find importing from the Kindle app hard to do, I suggest you instead download books from the manage my content page on amazon.com. That is the workflow I use personally. Once you have imported your existing collection, in the future importing a couple of books a week (at most) is a snap.

[eschwartz]

Personally, I think it makes a lot of sense to use the auto-add and have calibre delete the already-added books.
How else does calibre keep track of which files have already been auto-added?

You can set your Amazon account to automatically queue new books for Kindle for PC/Mac. Launch Kindle for PC/Mac and new books will be auto-downloaded. Launch calibre and they will be auto-imported.

This works fine unless you actually think Kindle for PC/Mac is a useful reader application instead of just being a tool to easily download your purchases. In that case, I cannot help you...

...

Of course, if Amazon would fix their software to not use retardedly obfuscated filenames (or reveal a public API allowing calibre to interface with your Amazon account ), this problem wouldn't exist.
But there is very little chance of that happening, and there is nothing calibre can do about it.

[emmgee]

I'm not qualified to comment on whether Finder and the Kindle app are "very poorly designed", but I'll stick my neck out and say that neither are "random". I would also say that an application that is a manager of digital books would by necessity need to accommodate the Kindle app – and an application that runs on a Mac would by necessity need to accommodate Finder. Lastly I will say, with respect, that Calibre is very complex and could be much more simple and achieve many of its design goals. That comment will may mean I won't get much more help in this forum but that's my opinion.

[emmgee]

The way that Calibre can remember which books it has imported without deleting is exactly the way that it detects duplicates when you select "Add books from directories, including sub-directories". It reads through all the books in your library and reports on the duplicates. It could just as easily read through all the books and add any books that are flagged as duplicates.

[eschwartz]

Originally Posted by emmgee

I'm not qualified to comment on whether Finder and the Kindle app are "very poorly designed", but I'll stick my neck out and say that neither are "random".

AFAIK Finder is doing nothing wrong, it is simply obeying Apple's filesystem hierarchy. Perhaps Kovid dislikes it for other reasons -- I don't touch Mac so I wouldn't know the difference.
I will happily agree that Kindle for Mac/PC is both poorly designed and random.

I would also say that an application that is a manager of digital books would by necessity need to accommodate the Kindle app – and an application that runs on a Mac would by necessity need to accommodate Finder.

And what exactly do you expect calibre to do about it?

You stated two problems:

1. Kindle for Mac stores downloaded files in a folder that Apple treats as hidden.
2. Kindle for Mac stores downloaded files with obfuscated filenames (actually, it is the Amazon product ID not that that is helpful).

Both of those are the responsibility of Amazon to fix. Once again, there is nothing calibre can do about it.

Lastly I will say, with respect, that Calibre is very complex and could be much more simple and achieve many of its design goals. That comment will may mean I won't get much more help in this forum but that's my opinion.

calibre is complex because it does many things in one application bundle. If you intend to claim that those things can be accomplished in a simpler way, I won't argue with you -- as long as you can provide solid reasoning and details. It is very easy to cast vague aspersions, and I suspect you are just casting vague aspersions.

By all means, prove me wrong. Bring on the details.

Note: "achieve many of its design goals" is not an option. "achieve all of its design goals" is a hard requirement... on account of that is the point of a design goal.

[eschwartz]

Originally Posted by emmgee

The way that Calibre can remember which books it has imported without deleting is exactly the way that it detects duplicates when you select "Add books from directories, including sub-directories". It reads through all the books in your library and reports on the duplicates. It could just as easily read through all the books and add any books that are flagged as duplicates.

But that is a stupid idea.

Because first it would need to do 99% of the work importing the book again, in order to determine if it is a duplicate.
And you just proposed doing that in a constant loop, to boot.

You REALLY didn't think this through...

[emmgee]

One sad thing about online discussions is that people find it hard to disagree without being disagreeable. I provided some feedback, not "vague aspersions". I made a suggestion and it was called "a stupid idea". Isn't there a better way to have a discussion than to lower the tone to name-calling?

I have worked in high tech for many years. I am very familiar with the design process. I have my ideas and my preferences. I put out a few thoughts in this thread. Why not? Isn't that what discussion forums are for? Do I need to make a rock-solid case for my thoughts to avoid being called "stupid"?

I like Calibre and I'm glad I entered my library into it. I'm glad, through this forum, to have learned a few things that will make my use of Calibre more productive. I haven't contributed yet but I will. I think if I get value out of a software product that the person/company who creates and supports it should get a fair amount of money in return.

I do think Calibre is overly complex. I would guess that more people would find it useful if there were fewer features and it was more simple to use. Maybe two versions would be a good idea, one fairly basic and one that has fuller features.

I'm sorry that you feel my ideas are stupid. I'm also sorry that you can't be more civil in your replies.

[emmgee]

In direct reply to the following:

Originally Posted by eschwartz

But that is a stupid idea.

Because first it would need to do 99% of the work importing the book again, in order to determine if it is a duplicate.
And you just proposed doing that in a constant loop, to boot.

You REALLY didn't think this through...

Try this: Download new books via the Kindle app then in Calibre select Add books > Add books from directories, including sub-directories (Multiple books per directory, assumes every ebook is a different book).

Calibre then does what you apparently think is a stupid idea. A pop-up appears that says "Reading metadata and adding to library" and presents a progress bar and titles that it is adding to the library.

Then it shows you the duplicates it found and asks if you really want to add them. If you click on "Select none", the duplicates are unchecked. Click "OK" and the non-duplicates are added.

My "stupid idea" is that since this functionality is already built in, it should be a fairly simple thing to change it this way:

Create an option under "Add books" that reads "Add new books". User clicks this option and Calibre does the same action as above but it assumes that the user does not want to add duplicates, only new books.

I hope you find this clarification less stupid. BTW, there are no loops involved.

[theducks]

Originally Posted by emmgee

One sad thing about online discussions is that people find it hard to disagree without being disagreeable. I provided some feedback, not "vague aspersions". I made a suggestion and it was called "a stupid idea". Isn't there a better way to have a discussion than to lower the tone to name-calling?

I have worked in high tech for many years. I am very familiar with the design process. I have my ideas and my preferences. I put out a few thoughts in this thread. Why not? Isn't that what discussion forums are for? Do I need to make a rock-solid case for my thoughts to avoid being called "stupid"?

I like Calibre and I'm glad I entered my library into it. I'm glad, through this forum, to have learned a few things that will make my use of Calibre more productive. I haven't contributed yet but I will. I think if I get value out of a software product that the person/company who creates and supports it should get a fair amount of money in return.

I do think Calibre is overly complex. I would guess that more people would find it useful if there were fewer features and it was more simple to use. Maybe two versions would be a good idea, one fairly basic and one that has fuller features.

I'm sorry that you feel my ideas are stupid. I'm also sorry that you can't be more civil in your replies.

9 posts and you know all about Calibre and how it works, and the many pitfalls

Calibre grew to what you see today. Go read the History of Calibre.

The Idea was poorly conceived due to a failure of the originator to fully understand the ramifications of repeatedly trying to import the same books.

There, is that a better way of saying that it is a Stupid Idea?

[eschwartz]

Originally Posted by emmgee

One sad thing about online discussions is that people find it hard to disagree without being disagreeable. I provided some feedback, not "vague aspersions". I made a suggestion and it was called "a stupid idea". Isn't there a better way to have a discussion than to lower the tone to name-calling?

"Calibre is very complex and could be much more simple and achieve many of its design goals" is feedback in the form of a vague aspersion that calibre is ill-designed.

And as I said above, I am more than happy to listen to what you have to say, as soon as you say it -- implies giving a reasoned explanation regarding the complexity you see and why and how you would make it simpler.

I will freely acknowledge I jumped the gun in assuming you had nothing to say other than what you said.
Although I have seen many people come here and say the same thing, and they always turn out to be cranks...

...

I don't consider myself to have lowered the tone to name-calling.
I consider myself to have attacked the merits and practicality of your one suggestion, and then concluded it was a stupid idea. Harsh? Maybe. Name-calling? Not at all.

I'd like to reiterate that past experience with people claiming calibre is needlessly complex inclines me to not think very highly of the suggestion.

I have worked in high tech for many years. I am very familiar with the design process. I have my ideas and my preferences. I put out a few thoughts in this thread. Why not? Isn't that what discussion forums are for? Do I need to make a rock-solid case for my thoughts to avoid being called "stupid"?

You don't need to make a rock-solid case, but you do need to make one that doesn't immediately get me to think "that would be a tremendous disaster of epic proportions".

I do think Calibre is overly complex. I would guess that more people would find it useful if there were fewer features and it was more simple to use. Maybe two versions would be a good idea, one fairly basic and one that has fuller features.

calibre already has the capability to e.g. hide unused toolbar icons and unused panels. Much of the advanced functionality is hidden behind menus or right-click options.

Making a second application as a lightweight alternative would be a lot of work, and I don't think Kovid is interested. He is happy with the current userbase which is quite large already.

Originally Posted by emmgee

In direct reply to the following:



Try this: Download new books via the Kindle app then in Calibre select Add books > Add books from directories, including sub-directories (Multiple books per directory, assumes every ebook is a different book).

Calibre then does what you apparently think is a stupid idea. A pop-up appears that says "Reading metadata and adding to library" and presents a progress bar and titles that it is adding to the library.

Then it shows you the duplicates it found and asks if you really want to add them. If you click on "Select none", the duplicates are unchecked. Click "OK" and the non-duplicates are added.

calibre does what I think is a very good idea actually. I suspect that is because you didn't actually internalize that i said it was a bad idea in the context of a discussion about auto-import.

Adding the books you specifically selected (recursively adding a specifically selected directory ) is a very reasonable thing to do though. Given that it was manually told to do so, and all.

My "stupid idea" is that since this functionality is already built in, it should be a fairly simple thing to change it this way:

Create an option under "Add books" that reads "Add new books". User clicks this option and Calibre does the same action as above but it assumes that the user does not want to add duplicates, only new books.

I hope you find this clarification less stupid. BTW, there are no loops involved.

Are we still talking about auto-importing books? Because manually importing them requires manually doing so, oddly enough, and one extra click is not my idea of a burden. In fact, I don't know why you are all of a sudden worried about the need to personally confirm adding duplicates. Adding duplicates is not expected behavior -- it is anomalous, and you should subject it to scrutiny.

Assuming you are still talking to me about auto-adding (feel free to ignore me if you secretly switched topics)... let me spell it out for you a little clearer.

calibre watches for changes in the auto-add folder using a QFileSystemWatcher.
When it gets a signal that the auto-add folder has changed, it recursively starts adding ebooks.
Leaving those ebooks in place means any ebooks which have already been added, get re-added every time the Auto-Adder triggers.
Given that the Auto-Adder is a constantly-running background process, the auto-adding happens in a loop. As in, again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again and again...

AFAIK there is no robust way to trigger on all modified files in a recursive tree, then remember that state after calibre is shutdown and restarted, catching files that were modified/added while calibre was NOT running, recognizing renamed files with byte-identical contents...
So recursively adding on a DirectoryChanged notification it is.

...

Or as theducks said, "a failure [...] to fully understand the ramifications of repeatedly trying to [auto-]import the same books".

[BetterRed]

@emmgee - what I and many others do for adding new books is something like the following

• import (add) books into a "New Books" library *
• do any necessary adjustments to basic metadata - ie Author, Title, ISBN and perhaps Series
• use the Find-Duplicates plugin->Find library duplicates feature to check for duplicates in the main library
• if I want to see details of what's in the main library I start calibre-server against the main library and access it via a browser.
• the books which are not duplicates I move to the main library, using Copy to Library (delete after copy).
• I resolve the duplicates either by
  
  • removing the book from the New Books library,
  • manually adding the format file (or replacing the existing format file) in the book in the main library - I do that by dragging the file out of the New Books book folder, and dropping it onto the relevant Book Details panel in the main library - calibre always does this as a copy. Then I remove the book from the New Books library
• At the end of the process I should have an empty New Books library

BR

*I normally do that either via Windows Send To, or by by dragging the format files from my file manager into the Book List.

[emmgee]

eschwartz, I submit... You are obviously of vastly superior intelligence to me. I withdraw my comments about the software perfection that is Calibre.