Annotating PDFs in OS X w/o modifying the file

[piovac]

I came out with a nice method to read and annotate PDF books in OS X without modifying the original files. The problem is that the default Preview.app saves all the annotation in the original files, which has the advantage to be portable to other PDF application, but has the disadvantage of continuously modifying the original PDF file.

A nice a alternative is to use Skim (http://skim-app.sourceforge.net/). I used it for years while working as papers reviewer and it really work nicely. By default Skim saves the annotation in the file extended attributes of HFS+. Alternatively, one can save files and notes in PDF Bundle (actually a folder), containing both the original PDF file and a file with the annotations. In any case, the original PDF file is untouched. Of course the application has features to embed and the embed the annotation in the PDF file in case you want to pass them to another PDF application, and the possibility to export the annotations in a separate file.

Using the default method (HFS+ extended attributes) is not convenient because the attributes get generally lost when the file is emailed, backed-up, etc. Moreover the import into Calibre strips the attributes away. For general use the creation of a PDF Bundle is my preferred. The problem is that the Calibre do not handle file formats that are actually folders and not real files.

The method that I came out with the better suit the use in Calibre is to use the external annotation file and (semi)-automate the process to import and and save the annotations. Here is the step-by-step process.

Step 1 - Install the Skim app if you do not already have it.

Step 2 - Associate the PDF files to the Skim. This is the only way to make it work with Calibre as it uses the default application provide by the system.

Step 3 - You can now read and annotate your files in calibre with Skim. When done, export the annotations in a .skim file: File->Export... and choose the Skim format, but do not modify the filename. This will save the .skim annotation in the same folder of the PDF file in the Calibre Library. Very important: DO NOT save the PDF files when you exit the program.

Step 4 - Next time you reopen the file Skim will ask you if you want to load the notes from the .skim file. You of course would say yes. And you can repeat step 3 when done.

Step 5 - The first automation task would be to load the annotations from the .skim file automatically without asking any question. This can be easily achieved using an hidden default option of Skim. To automate the loading operation by default, just open a terminal and type the following command:

defaults write -app Skim SKReadMissingNotesFromSkimFileOption -integer 1.

Step 6 - The export of the notes on a file and the closing of the PDF without saving is a very cumbersome operation with a lot of menu selections and click. But this can be easily automated with an Apple Automator service written in AppleScript and Assigned to a keystroke combination.
6.1- Open Automator and start a new service named "Save Skim Notes and Close.workflow"
6.2- On top of the windows select "Service receives [no input] in [Skim.app]"
6.3- For the action choose "Run AppleScript" and load the content of the script as follow then save it and the service is ready to use:

on run {input, parameters}
(* Your script goes here *)
try
tell application "Skim"
set filepath to path of front document
end tell
set filepath to POSIX file file path
set filepath to Unicode text 1 thru -5 of (filepath as Unicode text) & ".skim"
tell application "Skim"
save document 1 in filepath as "Skim Notes"
end tell
tell application "Skim"
close document 1 without saving
end tell
display alert "Skim notes saved on .skim file." giving up after 2
on error
display dialog "Could not save Skim notes!"
end try
return input
end run

6.4- Now you can assign a keystroke combination to the service (it will be under the "General" category). I chose Ctrl-Shift-Cmd-S as it is not used by any other Skip command.

6.5.- Ready to go. When Skim is open and you finish the annotation just press your keystroke or from the menu "Skim->Services->Save Skim Notes and Close" and the service saves the annotations file and closes the PDF without saving it.

Enjoy!

PS. Here are some links for Skim, Automator, etc. :
http://sourceforge.net/apps/mediawik...en_Preferences
http://sourceforge.net/apps/mediawik...le=AppleScript
http://veritrope.com/tech/the-basics...-applescripts/

[piovac]

The script has an extra space. Please change the line:

set filepath to POSIX file file path

with:

set filepath to POSIX file filepath

[BetterRed]

@piovac If I understand this correctly you are putting the .skim files in the Library/Author/Title (Book #)/ folders directly via a third party program.

There are various situations whereby calibre will/may lose those files - change the author name or book title, library check, library restore etc.

However if you add (register) the .skim files to calibre via the calibredb addformat command then calibre won't lose them. You can add (register) a file that's already in an existing book folder. What's more you see that it exists because SKIM will show up as a format in the Book Details panel along with PDF.

I assume this will will work on OS/X, I do something similar on Windows.

BR

[piovac]

@BetterRed You understand correctly, thanks for the tip. I was not aware that of the problem. I tried to copy the record to other libraries and the notes file was preserved. But then I just tried to rename the author and the file was lost.

I've other similar situations, described below, that probably need the same attention. Comments on how you handle this?

Supplemental material. I like to keep additional material like example code, errata, etc., including my scanned hand notes, together with the book. The method I found so far in OSX is to put all together in a ISO image and import it as a book format. Differently from the Skim notes, the "dummy" .iso book comes up in the list as a format. Does it mean that the ISO is at this point a registered format?

Paper books. Similar situation. I like to use calibre to keep record of my paper books. I created several dummy files (dummy.paperback, dummy.hardcover, ...) that I import when I create the book record and they show up as a format. Similarly as before, is it good or do I need to register the format as well?

One last question, is the formats registration data store in the calibre database of the library?

--piovac

[piovac]

Now I got. It is the same thing if I use the GUI and import it as book format.

I need to find a way to import add the format to book record the first time the notes got saved. The manual way would be to add a dummy file .<whatever format> before I start using it. But it would be nice if it would be recognize automatically once it is in the folder.

[eschwartz]

Under Library Maintenance ==> Check Library, I think there may be an option to add the extra formats to the book records.

[BetterRed]

Originally Posted by piovac

Now I got. It is the same thing if I use the GUI and import it as book format.

I need to find a way to import add the format to book record the first time the notes got saved. The manual way would be to add a dummy file .<whatever format> before I start using it. But it would be nice if it would be recognize automatically once it is in the folder.

Firstly, I think you may have read something into my earlier post that I didn't intend.

My bad - my use of the word 'register' can be interpreted as meaning - tell calibre to put a new format to into its internal list of formats it knows about - basically the ones that shown in the Preferences->Behaviour->Use internal Viewer for: list.

However that's not what I meant - I meant that the add_format subcommand of calibredb adds and records (registers) the existence of a 'format' file for a given book.

The calibredb add_format command does what the GUI Add Books->Add files to selected book record feature does. I don't have a Mac at hand so I don't know how that dialogue appears in OS/X - in Windows its a bit awkward to use so I went for what I knew would be the same

Calibre records (nee registers ) a list of what formats are present for each book in its database. It's when that list and the actual contents of the book folder get out of synch that unregistered format files are at risk of getting lost. As eschwartz has pointed out this can be 'fixed up' via the Check Library function.

However I think there may be a couple of better ways - not having a Mac, and not being you, I'm not sure what would the better way

This is what I would do in Windows to get the initial SKIM file registered

After opening the PDF in Skim (sadly no Windows version for me to test) I would make a brief note, save it and close the PDF/SKIM. I would then open the book folder from calibre (press 'o') and use the Send To context menu item to 'send' the SKIM file to calibre - if the Add Books settings are as per the attachment, then calibre will register the existence of the SKIM file into the book record in the database alongside the PDF.

I have configured Send To to use the calibre command, the syntax of which is calibre [opts] [path_to_ebook]. It turns out that the path_to_ebook can be any file, including file types that calibre doesn't know about (eg a SKIM file, or an AVI or an ISO). So Windows Send To executes the command calibre "path_to_file_selected"

But AFAIK OS/X doesn't have a direct equivalent of the Windows Send To feature, I found this http://reviews.cnet.com/8301-13727_7...ption-to-os-x/ which may help if you wanted to explore this method.

But I think this may be better way for you to explore

You could put a question in your Save Skim Notes and Close Automator script that asks - "Do you want register the SKIM to calibre?", if the answer is yes then issue a calibre or calibredb add_format command to register the SKIM to the calibre database. It may not be necessary to ask the question and to just do it in every save - but I'm not sure what the ramifications of doing that might be. Have a play with idea, I've no doubt you can work out what would work best for you.

BR

[BetterRed]

Much of the following reflects what I used to do regarding additional but I think I now have a better scheme which you can read about here Additional materials related to a book

The .7z approach works fine, and it has some advantages, but I think my latest scheme is more flexible - it also means I manage additional 'stuff' for authors and books in much the same way.

BR

Originally Posted by piovac

Supplemental material. I like to keep additional material like example code, errata, etc., including my scanned hand notes, together with the book. The method I found so far in OSX is to put all together in a ISO image and import it as a book format. Differently from the Skim notes, the "dummy" .iso book comes up in the list as a format. Does it mean that the ISO is at this point a registered format?

See previous post re 'register'

To add the ISO (for supplemental material I do the same as you but with .7z archives) you will need to use Add Books->Add files to selected book records, or the calibredb add_format command or the calibre [ebook_file_path] command.

One of the downsides of using an archive is that they have to be unpacked (in windows to a temporary area) to access the contents, and adding or removing content or updating something like a spreadsheet is 'fiddlesome'.

So if the information is volatile I keep it in a folder adjacent to but separate from the Library - ie alongside my Main library folder I have a MainInfo folder. The sub folders in MainInfo are a partial subset of the Main library subfolders. If I have anything in a MainInfo author folder I put a URI link to the folder in the Author Link field (see Manage Authors) and for a book I put a URI link to the folder into the My Comments/mycomments custom column. This means I can click Author Name in Book Details to open a MainInfo/author folder, and I can click the link in My Comments, which I display in Book Details, to open a MainInfo/author/title folder

Why My Comments and not the built-in Comments?

1. you can't edit Comments from the Book List (ie in a cell) but you can edit Long Text custom columns in a Book List cell.
2. if you use metadata downloading Comments might be overwritten if one isn't careful.

Originally Posted by piovac

Paper books. Similar situation. I like to use calibre to keep record of my paper books. I created several dummy files (dummy.paperback, dummy.hardcover, ...) that I import when I create the book record and they show up as a format. Similarly as before, is it good or do I need to register the format as well?

I use Add Books->Add Empty Book. You can maintain all the metadata in the normal way, maybe have a custom column to indicate you have the printed book as opposed to a book that you don't have in any form.

No need to create a dummy formats - calibre will create the author and book folders but all it will have in it is the opf file, you could use the download metadata to source covers, comments etc .

I think I've covered the 'register' issue - again apologies for the confusion.

BR

[piovac]

I wrote a complete applescript program that save the notes to a "sidecar file" and it also register this file to Calibre so it doesn't get lost. You can save the script in the Skim script library /Users/<username>/Library/Skim/Scripts

=====

(*
Save skim Notes to Calibre and Close
Skim script, written by Alessandro Piovaccari

ABOUT
This script save PDF annotations added with Skim in a separate skim file in the same folder of the .pdf
If the .skim format is not registered in Calibre for that entry, then it adds it to the database.

HISTORY
- 2015-01-17 – v1.0; first working version

KNOWN ISSUES
- File names that start with "_" can create a problem.
- If filename contains "_" than only the part of the filename till "_" is matched (short title)
This happens when the title has illegal filename characters that get substituted by "_"
- There could be a problem if there are 2 title with the same short title

TO DO
- Should check for file existance and ask for overwrite confirmation
- Skim format should be added only if format has not been register before
- Improve the filename matching
- I could add a confirmation before registering (skim file is saved correctly as it is at the same path of the PDF)
*)

--Set these to the correct path (these are for default installation)
set calibreCmd to "/usr/bin/calibredb"
set awkCmd to "/usr/bin/awk"

tell application "Skim"
activate
set filepathPdf to path of front document
end tell

set filepathSkim to Unicode text 1 thru -5 of (filepathPdf as Unicode text) & ".skim"

tell application "Skim"
save front document in filepathSkim as "Skim Notes"
close front document without saving
end tell

set file_info to the info for filepathPdf
set file_name to the name of file_info

--Get the title of the book
set titleOffset to (offset of " - " in file_name) - 1
set bookTitle to (text 1 thru titleOffset of file_name)
set undscrOffset to (offset of "_" in bookTitle) - 1
if undscrOffset > 1 then
set bookTitleShort to (text 1 thru undscrOffset of bookTitle)
else
set bookTitleShort to bookTitle
end if
--display alert bookTitle
--display alert bookTitleShort

--Get the author of the book
set authorOffset to titleOffset + 4
set bookAuthor to (text authorOffset thru -5 of file_name)
--display alert bookAuthor

--Determine the Calibre library path
set libraryOffset to (offset of bookAuthor in filepathPdf) - 2
set bookLibrary to (text 1 thru libraryOffset of filepathPdf)
--display alert bookLibrary

--Get the book record
--set calibreSearchCmd to calibreCmd & " list --library-path \"" & bookLibrary & "\" --separator=\";\" --search='author:\"" & bookAuthor & "\"' | grep " & (quoted form of bookTitleShort)
--set bookRecord to (do shell script calibreSearchCmd)
--display alert bookRecord

--Get the book ID
--The matching is done by doing a calibredb search per author and then the line with matched "short" title is considered the recors
--There could be a problem if there are 2 title with the same short form
--I could add a confirmation before registering (skim file is saved correctly as it is at the same path of the PDF)
set calibreSearchIdCmd to calibreCmd & " list --library-path \"" & bookLibrary & "\" --separator=\";\" --search='author:\"" & bookAuthor & "\"' | grep " & (quoted form of bookTitleShort) & " | " & awkCmd & " -F \";\" '{print $1}'"
set bookId to (do shell script calibreSearchIdCmd)
--display alert bookId

--add the format
set calibreAddFormatCmd to calibreCmd & " add_format --library-path \"" & bookLibrary & "\" " & bookId & " " & (quoted form of filepathSkim)
try
do shell script calibreAddFormatCmd
on error
display alert "Something went wront wile adding the .skim format to Calibre."
end try