Many academics now read journal articles on screen, as PDF files. Holdouts—and they are everywhere—print out forest-sized stacks of paper that teeter on crowded desks.
Freedom from clutter is just one advantage of digital reading. Another is searchability: a large database of articles is a crutch for our fallible memories. Thanks to Spotlight, a full-text keyword query can substitute for a laborious hunt for literature. A few carefully phrased searches is all it takes for instant recall.
Spotlight has a maddening flaw, however. Standard PDF annotations—both “sticky notes” and text “typed” on a page—are not indexed. As far as Spotlight is concerned, they do not exist.
For me this is no trivial problem. More often than not, these annotations contain the very text I’m searching for.
Papers, my reference manager of choice, has a partial solution. The software has its own annotation feature built-in. Though sticky notes created in Papers don’t get indexed by Spotlight, they are searchable with Papers’ own search function. Still, I often want to search PDF notes that aren’t annotated in Papers. I’m also wary of relying too heavily on Papers’ annotations since the feature is still a little buggy.
Thankfully there’s a better solution: Skim, the open source PDF software for Mac. Skim is designed for academics, integrates with LaTeX and BibDesk, and has great AppleScript support.
Skim is also Spotlight-friendly. The app stores its annotations in a separate file (with a .skim extension) that Spotlight happily indexes. And it’s easy to convert the Adobe-style annotations used by Preview, PDFpen and most other PDF software into Skim notes. Select “Convert Notes…” under the File menu, or assign a keyboard shortcut.
I now designate Skim as my default PDF editor in Papers. (In the “Paper” menu, scroll down to the “Open PDF with…” option. If you want Papers to default to Skim, rather than open a new tab in Papers, go to Preferences, select the “Papers” pane, and select Skim in the “Open PDF Files” drop-down.)
My problem was mostly solved, but what about all those PDFs already annotated using the standard tools? And then there were the PDFs annotated on the iPad. All of the best PDF apps on iOS—including iAnnotate, GoodReader, PDFpen for iPad, and (my favorite) PDF Expert—use standard, Adobe-style annotation tools. It takes too much time to manually convert each and every old or iPad-derived PDF to Skim formatting.
There’s an obvious need here for automation, but unfortunately no simple solution. The problem is that there’s no easy way to automatically detect that a PDF contains annotations (and therefore needs converting to Skim). None of the PDF metadata registers that a PDF has been annotated, so there’s no way to trigger an automated conversion.
Be warned that the solution I hit upon is fiddly. It works, but only justifies the set up if you have a large batch of PDFs to convert and/or if you frequently take notes on your iPad. The solution involves Hazel, the indispensable Mac automation tool.
A post on Hazel’s forums (by AppleSuperlatives) suggested using grep, the Unix plain-text search utility, for a related issue (detecting whether a PDF had been OCRed). The basic insight was that PDFs are, at core, binary files, which grep can search.
The next step was to isolate a snippet of text that gets added to a PDF’s binary when annotations exist. To do this I opened two PDFs—identical, except that one had annotations—in a text editor and saved them as text files. Next, I used the superb file-comparison tool Kaleidoscope to call out the differences between the two files. The annotations, in this case, had been made in Preview, and I found and copied some text (“Annot /T”) that gets generated in every annotation. This would be my trigger.
I set up a new Hazel rule to monitor my main Papers folder. (For basics on Hazel, see the tutorials and tips on the developer’s site.) The updated version of Hazel has a few new “if” conditions, including “Passes shell script.”
I embedded a short script to detect if the “Annot /T” phrase appeared; if so, the Hazel action would trigger.
Just in case I added a second condition, to ensure that the matched file is a PDF.
The next step was to ask Hazel to convert the standard-annotation PDF to Skim. First I prompted Hazel to open the file in Skim, and then added a short, embedded AppleScript.
I also added a separate Hazel rule, to run the conversion rule on all the subfolders in my Papers directory.
Soon all of my old, Preview-generated annotated PDFs converted to Skim. I encountered a problem, though, with the PDFs I annotated in PDF Expert for iPad: the “Annot /T” did not appear. So I used Kaleidoscope to find another phrase that appeared in PDF Expert-generated PDFs: “/Name /Comment”. I then modified the Hazel rule, using a nested condition—so that either “Annot /T” or “/Name /Comment” would trigger Hazel.
Now the articles I annotate on my iPad get converted too, once they’re back on my Mac. Of course if you use another iOS app like iAnnotate, you’ll need to locate a phrase and modify your Hazel rule accordingly.
It’s a hassle, for sure, but worth it. Now all my notes exist for Spotlight again. Memory restored.
You can also use Skim’s handy Convert Annotations menu command to convert existing annotations entered in other PDF readers
have you heard of writepdf? I wanted to give this one a try and it doesnt seem to bad at all
WritePDF for iOS looks strong, definitely. Feature rich at least, though PDF Expert has nearly all those features (including full-text search). Exceptions seem to be 1. Conversion to PDF (from, eg, Word file) and 2. Extended printing options (though this requires background app on Mac).
I’ve been making annotations with iannotate on my ipad, emailing the pdf to a sendtodropbox address through the app which extracts all highlights and comments into text of the email. The sendtodropbox extracts the text from the email into a separate txt file, puts the now annotated pdf in a folder that’s named after the subject line of my email, and I’m all done. I get a folder with two files, one the annotated pdf, the second, a txt file containing the text of my highlights and comments–which is, of course, searchable via spotlight. This is working nicely at the start as I can keep a clean copy of the pdf as well as my annotated one; but I’m unsure that it’s a long term solution as I’m relying on a webservice that might close at any time and I don’t think this is that viable with more than a few hundred pdfs.
I have a question, though it’s not directly related to the main point of your post.
Right now I am mainly using Adobe Professional to annotate my pdf articles. I sometimes do convert the Adobe annotations to Skim notes because I want to be able to use the Export notes function of Skim.
But what I find is that, after the note conversion, when I re-open the pdf file in Adobe, the annotations can no longer be seen in Adobe Does the conversion process make the annotation disappear or invisible to Adobe Professional? Is there a way to keep annotations visible for both Adobe and Skim?
You can always re-“embed” the notes, Adobe-style, to open in Adobe, Preview, PDFpen, etc, by choosing “Export” from the File menu, and selecting “PDF With Embedded Notes” (default). That should do the trick.
Great post. I have the same problem. I initially only used papers annotations but changed to skim to be able to edit annotations. Then I create some tags in the annotations (with a hashtag #) and the export to taskpaper if I need to use the paper for some specific research.
Your workflow seems great to convert a large amount of notes (though I’m gradually doing it manually). Great!
Now do you keep reading and annotating on the iPad? Is there any easier solution for the iPad workflow?
And how to do you manage the iPad mac/papers workflow?
Thanks a lot
And great solution!
Cool idea with the within-annotation tags. On the iPad I use PDF Expert, which employs the standard, Adobe-style annotations. I have my entire Papers folder–including nested folders of PDFs on–on Dropbox, though normally I import PDFs from the iOS Papers app. Once I’ve annotated a pdf, I move it to a Dropbox folder that my Mac sweeps using Hazel into my Mac inbox. After that, I normally replace the PDF in Papers for Mac with the newly annotated one, checking off the “read” box in the process. The script I describe in the post then converts the annotations to Skim notes. The iOS to Mac workflow, to recap, isn’t pretty. But at least one step, conversion to Skim notes, is now automated.
If you use iAnnotate it supports two-way dropbox sync so your annotations/highlights etc are already present in your copy in your Papers library. You still have to do the convert skim notes bit but it’s one less step for you and less duplicate PDFs on your system.
Thanks for the instructions. Strangely, although I am using PDFExpert on the iPad as you do, your search string didn’t work for me (at least not for an annotated file without any comments). I diffed the files and “/Annots” seems to be a search string that is able do differentiate the annotated and the original file, it also seems to work with files annotated with Preview. The bad thing is that it is still contained even if I remove the annotations in PDFExpert again…
Do you have any thoughts on that? Thanks, J
… please ignore my former comment – “Annots” doesn’t seem to be a good search term, it is contained in most of my PDFs (not the ones I first tested with).
Leaves the question: Do you know why your search term doesn’t work for me? Does it work for you with files that are only annotated not commented with PDFExpert?
This is a great thread, but really shows the next holy grail of academic workflows that needs to be found. Skim is a fantastic tool for note taking, but the lack of an iOS PDF reader that knows how to handle its notes is a real downfall. I keep my PDF library on dropbox and I have to use Skimnotes so I can maintain edits across multiple Macs…I really hope an iOS developer cracks this!
First off, thanks for a great article! I’m salivating at the awesome power of automation!
I’m with you, Mark. I use Goodreader and Skim to read through journal articles, synced with Dropbox. But, once I hit the ‘convert notes’ button in Skim, all my annotations disappear on the iPad. I’d love to find a way to have both.
An “advanced find” in Adobe Reader will search within the full text of multiple PDF file, including annotations. It’s a bit slower than spotlight though, despite building its own search index.
I think the main problem Skim’s resistance to follow the general PDF-annotation-flow. Every other app handles annotations in the “right” way (embedding them in the file). This should be the default in Skim and, on top of it, it could optionally apply its own annotation mechanism (why does converting annotations have to erase the original ones?). The current “exclusive OR” way in which it handles this (either Skim’s or Adobe’s way but not both) makes it a non-viable alternative for some people having the traditional workflow in mind, especially, as noted in the article and comments, for people annotating PDFs on their tablets.
True, although Spotlight doesn’t index traditional Adobe-style PDF annotating– and that’s the big issue for me. Half the value of making annotations is that they be searchable years later.
On a similar topic, you can automate the exporting of your Skim notes. I’ve written a script that exports all of your annotations into Markdown formatted plain text and embeds hyperlinks to the individual pages of your source PDF (if you use my custom URL scheme, or DEVONthink). For anyone interested, you can check out my blog at hackademic.postach.io
LikeLiked by 1 person
This is amazing and I’m considering adopting this method. However, since Papers 3 has been released, they’ve given us an option to synchronize our libraries to Dropbox for easier access between the Mac and iOS versions of the app. I say all of that to say that the library is stored on Dropbox so that it’s not an actual, browsable folder from within Finder, but a kind of archive!
So I wonder if this method will still work given that the script you’ve written here depends on the folder(s) being open and browsable? I’ll keep my eye on this (and perhaps try playing around with it…)
Thank you for this information. But there is a slightly better way of extracting the notes from a PDF annotated with a program other than skim. Instead of automating Skim with applescript you can run the command line tool that comes with Skim called skimpdf. So for example instead of the applescript would have the following shell script.
~/bin/skimpdf unembed “$1”
Have a look at the following, for where to locate skimpdf
Hope it helps someone
LikeLiked by 1 person
I’ve also been curious about PDF annotations and how to expose them to Spotlight. I guess the only way so far is to actually extract them. Skim looks promising, but I also came across this other (pretty) app. Not affiliated in any way and haven’t tried it myself but could be worth a look: http://highlightsapp.net
I am the developer of Highlights. I am a PhD candidate in biophysics and I actually learned Cocoa to scratch the very itch discussed here.
Highlights is a Mac utility for extracting PDF annotations (including images) to a range of formats (Markdown, HTML, PDF) or directly to Evernote, VoodooPad and DEVONthink. It can also fetch metadata automatically using DOIs and write the metadata to the PDF itself and lookup references underlined in the PDF (my favorite feature). The built-in PDF tools are limited, but aimed at letting you annotate on the PDF and add comments back using Markdown – keeping notes and PDF in sync.
Here are some promo codes, in case some of you want test the app:
Highlights looks like the bees knees! I’m trying to get org-mode org-ref and pdf-tools all working in emacs with a nested tree of pdfs organized in mendeley. Can’t make much progress and I need to do some real work instead. Would love a promo code. Congrats on your Ph.D. by now.
Hello everyone! If you need a free and reliable tool for converting pdf documents to text, mobi or epub formats, as well as URLS into pdf, I suggest to have a look here: http://kitpdf.com/ and see how it works.
LikeLiked by 1 person
Very useful. Has a more robust/simple process for doing appeared this since this post was written 2 years ago?
Wow. This is such a wonderful solution! I have been looking for this for years. Also…if anyone is using iAnnotate, the trigger I found was “/Annot”.
See Marc’s post above. I just used this to set up a Hazel rule on Dropbox to automatically embed Skim notes as PDF notes.
Pingback: Is there a way Spotlight indexes my Skim bookmarks/notes ? - PhotoLens