HELP -- Any efficient way to synchronize folders?

[mr.giroro]

I've been using Iliad for about 2 weeks. I am generally happy with it, but I've also found several (mostly "usability") issues. One problem is that there is no easy way I can synchronize document folder in Iliad and the collection of articles on my PC.

I am a researcher and have a large collection of PDF academic papers on my PC (7GB+). I bought a large CF (12GB), and was planning to synchronize the literature folder in my PC and the CF card (which will be used with my Iliad). I thought I could achieve this easily by using a synchronization software that I use to backup my files onto an external hard drive. The initial synchronization was easy--I just synced the literature folder with the CF card. However, I realized that once the PDF files are accessed by Iliad, Iliad creates a FOLDER and put the PDF file into the folder. Because of this, no further synchronization is possible because the file structures between my PC (all files) and my Iliad (all folders) are different.

The companion software can be used, but it is not really efficient to use it because I cannot simply add new files into the literature folder in PC and sync with the CF card. Am I missing something here?

I am wondering if someone has found a way to more efficiently synchronize Iliad and PC. If so, please enlighten me. It will be tremendously helpful. Thank you in advance.

[daudi]

I was initially stumped by this PDF-folder issue. I decided to keep my master copy of my library on my USB (CF in your case) and then back that up regularly. When I get a new article I download it to the PC (because that's what I use for searching) and decided to write a little script that:

i) creates a PDF container folder,
ii) writes a manifest.xml file
iii) includes an image of the front cover of the journal
iv) allows me to enter the title of the article so it appears nicely in the contentlister

I then move that container to my library on my USB. I then see the article with the full title and an image of the journal cover so it looks "pretty" and helps me to remember the article.

I was nervous about doing this at first, but I use rsync to do quick back up my USB key on a regular basis and it works fine for me. It is simple to plug the USB into my PC if I want to view an article on the PC or print it (but this is rarely necessary).

[grayfox]

Hi,

I copy from the CF card back to my PC ( a mac ) using rsync.

By copying from the CF card back to the PC the folder structure is
preserved.

You might want to check to see if rsync has been ported to the
PC ( windows).

With rsync the backup is extremely fast ( after the first).
the command I use is

#!/bin/sh
# Add --delete to delete files on the target that are not on the source
rsync -av --delete /Volumes/CFILIAD/ /Users/jerry/CardBackup

which is a one liner

Jerry

[mr.giroro]

Dear Daudi & Grayfox:

Thank you for your tips. All make a lot of sense. Yet, I wish there was a better solution. It would have been much better if "manifest.xml" and "scribble.irx" files are created in a separate hidden directory rather than creating all the separate folders. That is very inefficient in many ways. It is just too much work to create a new folder whenever I downloaded/scanned a paper (although I agree with Daudi that it has some perks (e.g., journal cover--a very nice touch)). On the top of that, it becomes rather annoying when I have to search and read papers on my PC (extra clicks to flip through folders). I hope iRex addresses this sometime in the future.

I would appreciate any other tip.

Thanks.

[daudi]

I seem to remember reading a post from a while back where someone else was also requesting a separate hidden folder for manifest files (possibly on the irex forum) and the responses where clear that the format was not going to change. I think there would probably be a greater risk of the additional info being disassociated from the PDF files and things could get very messy for some users, so I think the current approach is probably safer.

For me creating the container folder and manifest file is not hard work or time-consuming. I have a script that does it all for me that is integrated into my file manager so it takes barely a second to do it, certainly less time than it takes me to actually save the PDF in the first place. I believe that windows is more limited when it comes to scripting, but my script is pretty simple and it should be possible to reproduce it as a batch file or if not there is a version of bash.exe that runs on windows that could do it. I have no idea if it is possible to add a script/batch file as a right-click option in explorer to make it easy for regular use.

As for searching and reading: do you use a bibliography manager? Many now have a field for a link to a PDF copy of an article. You could have that link pointing to the PDF in the container folder and use your biblio software for searching and opening PDF files. (And what are doing reading PDFs on your PC now that you have an iliad!!! )

[mr.giroro]

Hi Daudi,

Creating hidden directories could get messier because, if Iliad creates a new directory for each PDF, the number of files/folders will be practically doubled. Yet, I think the benefits FAR outweigh the potential costs especially for those who have to manage 1000s of documents.

I use Endnote for managing my references. Endnotes also creates a directory for each PDF files linked to the software (and keep the PDFs in the respective directory). Now that I have an Iliad, I have to have Four different copies of the same documents in four different folders--two for Iliad (one for sync and the backup created by Companion software), One for Endnote, and one for the main collection. This not only wastes space on my hard drive (7GB x 4 = 28GB) but also requires a lot of my attention to manage them. It would have been much easier if I could simply sync my literature directory with Iliad.

Also, any further revision to my document collection/structure gets much more complicated. As I have to sync back and forth between Iliad and PC, this makes room for mistake and headache. I see there is a lot of room for improvement in Iliad's software (certainly for such an expensive piece of gadget). I hope the usability of the software gets better sometime in the near future. Otherwise, I may have to go back to my X60 Tablet and continue to destroy my eyes with the flickering screen

Thank you for your comments and suggestions.

[wallcraft]

Originally Posted by mr.giroro

However, I realized that once the PDF files are accessed by Iliad, Iliad creates a FOLDER and put the PDF file into the folder. Because of this, no further synchronization is possible because the file structures between my PC (all files) and my Iliad (all folders) are different.

It should be possible to charge the behavior of the ContentLister to NOT create a folder. This might loose some functionality, but I'm not sure how much.

The MobiPocket Reader already works this way. Usually it creates a new directory for each .prc file when it is first opened. However, it does NOT do this in the Books/Mobipocket subdirectory. I assume this is handled by the ContentLister. Note that this works in part because MOBI auxilary files have the same name but a different filename extension (and the contentlister can therefore ignore the auxiliary files).

[comtrjl]

Microsoft have released a fairly flexible file synchroniser called Synctoy - now up to version 1.4, I believe. You could give it a try and see if it does what you want. It doesn't cost anything.

[daudi]

Originally Posted by mr.giroro

[...]
I use Endnote for managing my references. Endnotes also creates a directory for each PDF files linked to the software (and keep the PDFs in the respective directory). Now that I have an Iliad, I have to have Four different copies of the same documents in four different folders--two for Iliad (one for sync and the backup created by Companion software), One for Endnote, and one for the main collection. This not only wastes space on my hard drive (7GB x 4 = 28GB) but also requires a lot of my attention to manage them. It would have been much easier if I could simply sync my literature directory with Iliad.

Also, any further revision to my document collection/structure gets much more complicated. As I have to sync back and forth between Iliad and PC, this makes room for mistake and headache. I see there is a lot of room for improvement in Iliad's software (certainly for such an expensive piece of gadget). I hope the usability of the software gets better sometime in the near future. Otherwise, I may have to go back to my X60 Tablet and continue to destroy my eyes with the flickering screen

Thank you for your comments and suggestions.

OK, I see your problem. iRex mucks around with the location of PDFs in one way, endnote mucks around with them in a different way. At first I thought that endnote was just pointing to an existing file because the menu option says "link to PDF" but it actually makes a copy and stores it in a sub-directory. At the risk of appearing provocative, perhaps it is endnote that should do things differently

I think, however, that there is a way of coercing things to work so that you could have a single copy of each PDF (and then back it up somewhere), and have easy access from endnote and be able to view the PDFs on the iliad using the contentlister to navigate. I think you could do something based on a kludge that I find useful. First I'll explain my kludge and then how you could use it.

I have a single folder on my USB key with all my articles in it as PDFs, each PDF in a container folder with manifest file. I name the PDFs in a consistent manner so I find it easy to find a particular reference if I know the first author. I can use the iliad search to find all articles by author xxxx, for example.

When I am working on a project or paper, however, I like to keep the papers that I am reading and using for that project with that project. I think of my full collection as a library, and the copies that I keep with a paper or project as items that I have borrowed from the library.

One of the many things I like about linux is symlinks, where you can refer to the same file by creating symlinks in different places. This goes beyond what windows shortcuts offer. On my linux box I used to create a symlink that links to the articles that are in a project.

The iliad uses linux, but by default USB and CF memory do not use the linux file system so symlinks don't work. But I found that it is possible to create a container folder with a manifest.xml file that works like a shortcut. I have created a script to do this for me, so now on my PC all I do is select a container folder on my USB that has the PDF article, hit a hotkey and it creates a new folder with a manifest.xml that points to the original PDF. It knows about the mount points on my PC and on the iliad and changes the path accordingly. I then move that "shortcut" to the appropriate place on the USB key for the paper that I am working on. When I use the USB key on the iliad and select the shortcut it opens the PDF at the original location. It even uses the same scribble file so any notes I made previously are visible and any new notes I make are available no matter where I open the PDF from.

Now, endnote stores the location it has saved the PDF to in the endnote database. I think therefore it should be possible to do the following:

i) export some fields from endnote, including the location of the PDF file to some text format that can be used in a script.

ii) process the exported information and for each reference to a PDF file create a folder and manifest.xml that points to the correct location of the PDF file. The paths would need to be corrected to allow for what they would be on the PC versus what they need to be on the iliad---that should be part of what the script does.

If you have your endnote file (and it's data folders) plus these "shortcuts" on your CF disk then you have all of your library in one place. When you use it on your PC you can search for and open the PDFs using endnote. When you use it on the iliad you can use the "shortcuts" that refer to the actual locations of the PDFs in your endnote data folders.

This way you use endnote as your master for managing your references and you just run the script whenever you need to create the shortcuts.

And then you can just back-up the CF card regularly with any kind of sync software (there is even rsync.exe for windows) just in case you lose it in a Swedish library.

[mr.giroro]

Dear Daudi:

Thank you for your amazing reply. I appreciate taking your valuable time to share your tricks and experience. I found your suggestion very helpful. Endnote is a very valuable piece of software, but I must agree that it is not a well-written software. There are so much room for improvement.

Your solution seems to be a great one; it will be great to be able to have only one set of references on my PC. I will try to learn your trick. I haven't used Linux, and need to educate myself.

[readmore]

There are some great ideas here - has anyone given some thought as to how to make this work with zotero? (www.zotero.org) Since Zotero stores its files locally (and can be run on a USB drive with portable apss firefox).... maybe one possibility could be having everything run off the USB drive only?

Taking this much further than my tired brain should right now- zotero (free software, you should take a look if you haven't seen it), uses a flat file system with sqlite. I assume the squiggles can be created in the same directory as the PDF automatically. Merging would be simply running through the PDF directories and looking for squiggles and merging them. Backup would be straightforward as backing up the USB drive. If you don't want them merged, editing the sqlite file indepedently is theoretically possible, or just manually adding it to the zotero database.

Manually finding articles on the USB drive may be a bit of a challenge, but some of the ideas posted here may resolve that side. Some type of automatic parser should be straightforward.

[mr.giroro]

My quest for a way to synchronize files in Iliad with those in my PC continues. After some more exploration, I am very close to conclude that it is IMPOSSIBLE to synchronize files between Iliad and PC.

The biggest problem is that Windows does NOT allow you to have a folder and a file with the same name. Consider the following common scenario for Iliad users.

You have a file "a.pdf" in your PC. You copied the file onto your CF card, and read it from your Iliad. Then, your Iliad creates a FOLDER called "A.PDF", and put "a.pdf" into the "A.PDF" folder and create manifest and scribble files in the folder.

Now you added a new file "b.pdf" in your PC, and want to synchronize the document folder in PC with the document folder in CF so that "b.pdf" is copied to CF, and "A.PDF" and associated files are synchronized to PC. The problem is that Windows does NOT allow you to have a folder and a file with the same name. That is, you cannot copy A.PDF from CF to PC because PC has a.pdf (a file with the same name). Windows programs will NOT write A.PDF over a.pdf. I tried several synchronization software, but found that none of the programs work.

So, I hit the wall. I am not sure how others handle this. Any thought? Thank you in advance for your advice.

[daudi]

Have you had a look at rsync? There is a DOS/windows version out there somewhere. It is very flexible once you get your head around it.

The other thing that occurs to me is that you might need to break this into two cycles of tasks/work. With subversion (and other) version control software you have a cycle of work:

i) update to get the latest version
ii) do stuff
iii) check in your new work

Translating that to your situation:

i) you sync from your PC to CF to get the latest copies on the CF
ii) you open the PDF on the iliad and scribble all over it
iii) you then sync from the CF back to the PC (before you download anymore that you want to sync to the iliad)

You say that your sync software won't do the third step. I'd be surprised if rsync won't. I can't try it now, maybe later.

[mr.giroro]

i) you sync from your PC to CF to get the latest copies on the CF
ii) you open the PDF on the iliad and scribble all over it
iii) you then sync from the CF back to the PC (before you download anymore that you want to sync to the iliad)

You say that your sync software won't do the third step. I'd be surprised if rsync won't. I can't try it now, maybe later.

Exactly! This is what I have been trying to do. I hit a road block in step (iii) because all the synchronization software refused to do the step (iii). If the folder in the source folder (CF) has the same name as the file in the destination folder (PC), Windows does not allow the folder to be copied into the destination folder. If the source and the destination folders have files with the same name, it is not a problem because any synchronization software will simply copy the newer file onto the older one. However, if one is a folder and the other is a file, Windows does not do that.

I tried to search for RSYNC for Windows, but couldn't find it. Even if it has the Windows version, I somewhat doubt if it would do this because it is really Windows' limitation.

I am wondering how others do this. Synchronization is crucial for people with many papers, and I cannot imagine how one can use Iliad without this functionality (especially scholars and researchers).

Thank you for your continuing advice and help.

[nekokami]

@readmore, Zotero needs Firefox, and Firefox is too resource-intensive to run on the iLiad.