Important news for iTunes/iDevice users: changes in 0.8.7

[GRiker]

Beginning with version 0.8.7, significant changes were made to calibre's database storage architecture in anticipation of supporting networked databases. As a consequence, some iDevice users may want to adjust their iTunes settings. All iDevice users should be aware of how this change affects books transferred from calibre to iTunes and iBooks.

Does this apply to me?
If you are using calibre to transfer ePubs or PDFs to iTunes and ultimately iBooks on an Apple iDevice (iPad, iPhone, iPod Touch) this applies to you.

What do I need to do?
You need to decide whether you want iTunes to store copies of transferred books, or if you want calibre to create a local copy of transferred books for iTunes to reference.

• Open iTunes Preferences|Advanced
  
  
• Choose a setting for Copy files to iTunes Media folder when adding to my library. If you already have existing content in iTunes (music, videos, etc), you should keep your current setting.
  • Checked: iTunes will make copies of transferred books in your iTunes Media folder
  • Unchecked: iTunes will reference copies of transferred books in your calibre configuration directory.
• Open calibre Preferences|Plugins|Device Interface plugins. Select the Apple device interface (it may be disabled, that's OK). Click Customize plugin.
  
  
• Set 'Copy files to iTunes Media folder…' is enabled in iTunes Preferences|Advanced to match the setting you chose in iTunes Preferences|Advanced above.
  Important: Both checkboxes should be in the same state (checked or unchecked). Mixed settings may lead to an inability for iTunes to sync your books, or multiple local copies.

I really don't like reading technical explanations. May I stop reading now?
If you've followed the previous instruction, you're done. If you're concerned about why this change was implemented, or its implications, read on.

Why do I need to do this?
Changes to calibre's database storage architecture mean that it is no longer possible for a device driver to directly access calibre's local copy of the file, as one may not exist. This change ensures that a local copy of the transferred file will exist on your system, either in your iTunes Media folder, or a special 'iTunes storage' folder in your calibre configuration directory.

What about books previously transferred from calibre to iTunes?
Books transferred from calibre to iTunes prior to 0.8.7 still have valid paths to calibre's local database, so iTunes should still be able to sync them with your iDevice. If you make a change to a book previously transferred, you will need to resend it to iTunes.

Doesn't this mean that there will be extra copies of transferred books on my system? Why can't things be the way they used to be in the good old days? Grumble grumble grumble.
Calibre will eventually support networked databases as an option. In a networked database, there are no local copies of the calibre library. When a file is sent to iTunes, a local copy needs to be available for iTunes to be able to sync it with your iDevice.

I don't want extra copies of my books floating around on my system.
If you are concerned about storage space on your system, you can complete the transfer to your iDevice during the same calibre session, then remove the local copy:

• Connect to iTunes by clicking the Connect/Share icon, then select the Connect to iTunes option.
• Select the book(s) to transfer, then click the Send to Device icon.
• Using iTunes, sync your iDevice to transfer the books to your iDevice.
• Re-select the books(s) just transferred, click the down arrow next to the Remove books icon, then select Remove matching books from device. The selected books will removed from local storage, and from the iTunes database.

What's the proper way to delete books added to iTunes/iDevices?
Books should be removed from iTunes/iDevices using the same method you used to add them:

• If you added the book to iTunes using calibre, delete it using calibre's Remove matching books from device command.
• If you added the book directly to iTunes, remove it using iTunes.
• If you added the book directly to iBooks on the iDevice (a download from the iBookstore, or using calibre's content server), delete the book directly in iBooks on the iDevice.

I have books listed in iTunes with a '!' prefix. What do I do?

1. In iTunes Preferences|Advanced, confirm that the setting for 'Copy files to iTunes Media folder when adding to library' matches the setting in calibre Preferences|Plugins|Device Interface plugins|Apple device interface as described above.
2. Resend the books shown with a '!' prefix

Is it OK to wait for a release or two in case there are problems with this change?
It is possible that some untested scenarios will cause unanticipated problems. Many users wait to upgrade until they see a new feature or bug fix in the weekly release notes of interest to them, or to see if new features create new bugs.

What do I do if I have a problem transferring books to iTunes/iDevices after updating to 0.8.7+ and following the directions above?
If you have updated to version 0.8.7 (or later), followed the directions above, and you're having a problem transferring books to iTunes/iDevices, reply in this thread with a description of the problem, including:

• Your OS and version
• Your version and nationality of iTunes
• The steps you took leading up to the problem
• The problem

[nessan]

This doesn't seem like a good idea.

Won't checking the 'Copy files to iTunes Media folder when adding to library' option in iTunes change how that program works with all audio and video files—not just the epubs from Calibre? Many people will not be interested in having all their carefully organized gigabytes of audio and video files copied in to iTunes just to make Calibre happy. Can some other scheme for exporting books to a well known place for iTunes be implemented instead please?

[MovieHQ]

Yes, I am pretty sure it will change it for all your media. I also do not want itunes to copy 200+ gigs of music to the itunes folder. Are there any other options?

[Stampercam]

I gotta say, this change is going to cause a lot of headaches for many users. I use itunes for a lot of different purposes, including video lectures from university, academic podcasts, music, audio books from audible.com and other. I have mine organised very specifically on my harddrive where I want things, and the way calibre handled the itunes imports before was perfect because it didn't impact on my other media types. This change will have DIRECT impact on all the other files I need to store with itunes. If another way could be found it would be much better.

[ApK]

I agree, this seems like a really bad thing. (Wow...first time I've ever said that about anything Calibre related!)
I think this may be the first time Calibre has necessitated a change to the way we use other stuff.
I, too, have my multi-gigabyte music and media library on a nas, and my wife's iTunes is only one of the several things it's used with.
I absolutely don't want to change that iTunes setting and duplicate everything we add on to her laptop.
Please tell me there is another way to handle this other than never updating Calibre past 0.8.6, or only having her load books to her iPod via Content Server?

[Misplaced Mage]

Originally Posted by GRiker

Beginning with version 0.8.7, significant changes have been made to calibre's database storage architecture in anticipation of supporting cloud storage.

GRiker, can you be more specific? Does this have to do with WiFi iCloud sync, or is there some other cloud storage scheme in the works for Calibre?

Without more information, I can only say that this change won't work for me, either. I, too, have hundreds of gigabytes of audio and video scattered across several hard drives and a NAS, and have no one storage volume currently large enough to handle the entire collection. I thus can no longer permit iTunes to copy incoming media files because it doesn't have provisions for automatically organizing a multi-volume library. The one saving grace at present is that I use a Kindle for reading, but I will likely switch to an iPad as soon as Apple boosts its display resolution; at that point this will become a serious problem for me.

[GRiker]

I'm working on a revision for 0.8.8 that should address these concerns, will post a description of how it will work here when I've tested it. If this is an issue that concerns you, stick with 0.8.6 for now.

G

[Cascades]

Just to clarify...
You can turn this feature on and off and it only affects what you are currently adding to the library - it won't change anything that has already been added.

For those of you who are really worried then only turn this on when adding eBooks, then leave it off the rest of the time, but it won't move all your files around unless you consolidate your library.

[GRiker]

Originally Posted by Cascades

Just to clarify...
You can turn this feature on and off and it only affects what you are currently adding to the library - it won't change anything that has already been added.

For those of you who are really worried then only turn this on when adding eBooks, then leave it off the rest of the time, but it won't move all your files around unless you consolidate your library.

Correct.

G

[drMerry]

Another -1.
I store my music, ebooks and video on different drives in my network. Changes in calibre should always be independent of other programs.

[GRiker]

Here's what I'm proposing for version 0.8.8:

I've added a configuration option in the Apple device driver customization dialog matching the similar iTunes configuration option.

The proposed Apple customization dialog for 0.8.8:

The tooltip when hovering the cursor over the new option reads "This setting should match your iTunes Preferences|Advanced setting. Disabling will store copies of books transferred to iTunes in your calibre configuration directory. Enabling indicates that iTunes is configured to store copies in your iTunes Media folder."

The iTunes Preferences|Advanced dialog:

The new recommendation will be to set these options identically:

• If you want iTunes to store your media, enable the option in both calibre Preferences|Advanced|Plugins|Device interface plugins|Apple device interface customization options and iTunes Preferences|Advanced. Enabling the option in both dialogs will cause calibre to pass the file to iTunes, which will make its own copy in your iTunes Media folder.
• If you don't want iTunes to store your media, disable the option in both calibre Preferences|Advanced|Plugins|Device interface plugins|Apple device interface customization options and iTunes Preferences|Advanced. Disabling the option in both dialogs will cause calibre to create a copy of each downloaded book in a calibre configuration subdirectory called 'iTunes storage', which will be where iTunes expects to find the file.

If you have an opinion about this, please add it to this thread. I'll be testing this implementation before the next release.

G

[ApK]

If I understand the above correctly, the alternative option will mean that Calibre will create a duplicate of any book we send to the iDevice?

Presumably that duplicate will need to exist as long as we care for iTunes to be able to use that item.

I wonder if it would be a good idea to have the option for that duplicate to be deleted when we close Calibre, or something like that, once we do the needed sync, so the added copy isn't laying around, and we could always resend it from Calibre if we need to resync it in iTunes again.
I'm not an Apple/iTunes guy-- itunes is only a necessary evil to me to get books on my wife's iPod--so I don't know if I'm making sense to everyone else, here.

BTW, can you point me to info on the architectural changes that are making this necessary?

tx,
ApK

[GRiker]

Originally Posted by ApK

If I understand the above correctly, the alternative option will mean that Calibre will create a duplicate of any book we send to the iDevice?

Yes

Originally Posted by ApK

Presumably that duplicate will need to exist as long as we care for iTunes to be able to use that item.

Yep.

Originally Posted by ApK

I wonder if it would be a good idea to have the option for that duplicate to be deleted when we close Calibre, or something like that, once we do the needed sync, so the added copy isn't laying around, and we could always resend it from Calibre if we need to resync it in iTunes again.
I'm not an Apple/iTunes guy-- itunes is only a necessary evil to me to get books on my wife's iPod--so I don't know if I'm making sense to everyone else, here.

Different folks have different sync strategies for their iDevices. The current implementation will delete the book from the local storage when the user deletes the book from the connected device, i.e. iTunes. That's the only safe way to manage deletions - let the user decide.

Originally Posted by ApK

BTW, can you point me to info on the architectural changes that are making this necessary?

That's a question that Kovid will need to respond to.

G

[ApK]

Originally Posted by GRiker

Different folks have different sync strategies for their iDevices. The current implementation will delete the book from the local storage when the user deletes the book from the connected device, i.e. iTunes. That's the only safe way to manage deletions - let the user decide.

OK, that makes sense. So with this new option, we can choose to either:
1. Let Calibre's send-to-iTunes copies remain where ever it is they are created (presumably we can choose that location) and it will essentially be a separate folder of books used in the iTunes library, or

2. We can delete the books in iTunes as soon as we are done putting them on our devices. This will clean up the extra copy, and we'll just have to send it from Calibre if we ever lose it from the device and need to re add it.

If I'm reading all that correctly, #2 sounds pretty painless for those of us who don't want iTunes to make copies of our files.

ApK

[Grizzly99]

I'm not trying to be rude with the question I'm about to ask--Calibre is one of the best programs out there...but what possible advantage is this change? I honestly don't understand why someone would want to have duplicate copies of their books/media on their computer.

Is this something mandatory from Apple?