Auto-add weirdness

[davidfor]

Apart from the difficulty of making this robust across platforms, I would feel uncomfortable with an application changing permissions of files. This is treading in murky waters and means changes to the system that aren't expected.

Also, I can guarantee that there is someone out there relying on this behaviour. Someone is putting read-only files in the auto add directory knowing that they won't be processed immediately. I have no idea why they are doing this, but I am sure that someone is.

[kovidgoyal]

Absolutely, auto-adding is already pretty magical (particularly on windows, with its crazy file locking semantics) -- however, the solution to something being magical is not to add more magic to it, it is to remove as much magic as is possible. Read the code for the calibre auto-adder and you will find that > 75% of it is to deal with problems like antivirus locking files, or files that are partial because they are in the process of being transferred from a slow source, such as a network. That is all work calibre does to make the process as predictable as possible, given the inherent limitations of filesystems and OSes. Speculatively chmoding files before trying to add them absolutely will not increase that predictability.

And yes, as davidfor points out, file permissions can be a signal just as much as they can be noise. I have to agree with him that having an application simply override file permissions is disconcerting to say the least.

[Phssthpok]

OK. So bottom line, it's not something for everyone (which is why I originally suggested making it a prefs option which I and others like me could turn on).

At the moment I have a low-quality solution (the batch file I described earlier) which pops up an annoying console window when it runs; I could rewrite it in in a real language to prevent the annoyance, but presumably the "right" solution would be to rewrite it as a plugin which would start and stop when Calibre starts and stop, and wake up every so often to do my nasty magic tricks...

So, would this make sense? If so, I'll probably start bugging people on the plugins forum for advice on how to do it, such as plugin lifecycle issues and how to get at the auto-add directory...

[kovidgoyal]

Seems awfully complicated. Why dont you just create a script that will extract the files from your archives with proper permissions instead, if you cannot find any program that does it for you. If you cannot figure out how to do it directly, then make it a two part operation, one that extracts the files to a temp dir, fixes their permissions, and then moves them over to the auto add dir.

Then just run your script instead of using whatever unarchiver program you are using currently.

[kovidgoyal]

Here's a one liner that will extract the contents of a zip file into the current directory.

calibre-debug -c "import sys, zipfile; zipfile.ZipFile(sys.argv[-1]).extractall()" file.zip

[kovidgoyal]

And here's a one liner that will mark all files int he current directory as writable (assuming chmod works)

calibre-debug -c "import os, stat; from calibre import walk; [os.chmod(f, stat.S_IRWXU) for f in walk('.')]"

[Phssthpok]

Originally Posted by kovidgoyal

Seems awfully complicated. Why dont you just create a script that will extract the files from your archives with proper permissions instead, if you cannot find any program that does it for you. If you cannot figure out how to do it directly, then make it a two part operation, one that extracts the files to a temp dir, fixes their permissions, and then moves them over to the auto add dir.

Then just run your script instead of using whatever unarchiver program you are using currently.

So, write a special-purpose program which lists the contents of a zip file, lets me select the files I want (because I usually don't want everything that's in the archive, so I also need a GUI), and then extracts them into files with the right permissions. And what about other formats (rar, 7z, and what-all else that people use)? It gets messy, it turns into a non-trivial project.

Plan B, using a temp directory and all that as both you and betterred have suggested means that I have to (a) extract the files, and then (b) manually run something to fix the permissions. Or have (b) run (a), wait for process (a) to terminate, then continue with the moving etc.

In either case, the other issue is I have to remember to run my special program instead of just double-clicking on an archive. I might as well not bother using auto-add at all -- just extract into a temp directory and then drag&drop into Calibre.

At the moment I have a solution which involves no extra effort on my part -- I just extract the files into the auto-add directory. Every 30 minutes a batch file runs that checks if there are any files, fixes the permissions, creates and deletes a file, and goes back to sleep. Like I said, I could rewrite it easily enough and run it from startup or install it as a service -- but then it would be running even when Calibre ain't. So the plug-in solution seems better.

I'm imagining a plugin that kicks off a thread which will basically just sit in a loop which, with a bit of hand-waving, looks something like this:

while keepRunning:
    gotfiles = False
    for file in os.listdir(gprefs['auto_add_path']):
        os.chmod(file,stat.S_IWRITE|stat.S_IREAD)
        gotFiles = True
    if gotFiles:
        flagFile = os.path.join(gprefs['auto_add_path'],'foo.xxx')
        open(flagFile,'a').close()
        os.remove(flagFile)
    time.sleep(loopDelay)

The big question is, is this feasible? Am I on the right lines?

[edit: Just caught up with your second one-liner, which would simplify the above code somewhat. Thanks.]

[BetterRed]

@Phssthpok - FWIW - if you put a shortcut to calibre.exe your Windows Send To folder then you can send a file with any of the common attributes set - readonly, system, hidden, archive. Works from a compressed volume, not sure about an encrypted volume.

You can send multiple 'books', if you want to pick and choose what to send, the check boxes in Windows file manager are useful. For me at least this is a 'better' option than Autoadd.

BR

[kovidgoyal]

You really need to embrace the power fo scripting. If you want a gui to extract the files, use one. Have your script launch a GUI program and wait for it to terminate. After it terminates have it fix permissions and copy the files. No extra effort on your part the and required script is five lines long.

[eschwartz]

Or simply add the magic directly to calibre via a source checkout.

As I understand the main objection is that it would run contrary to peoples' expectations.
So if you, personally, want calibre to speculatively chmod fils before trying to add them, then you, personally, can tell calibre to do so.


But I would feel nervous about adding it upstream -- what, more options, and this one to do something that is inadvisable to begin with?
There is such thing as feature creep, and features designed for one specific user are feature creep. Besides, the existence of an option would encourage people to use it!

[Phssthpok]

Originally Posted by kovidgoyal

You really need to embrace the power fo scripting. If you want a gui to extract the files, use one. Have your script launch a GUI program and wait for it to terminate. After it terminates have it fix permissions and copy the files. No extra effort on your part the and required script is five lines long.

Indeed, this was exactly the plan B I described earlier. Like I said, my only real objection to this is I have to remember to use the script to process the file rather than double-clicking and extracting. (And besides, I thought it might be fun to learn a bit about writing plugins.)

All I really want to know at this point is, are there any foreseeable problems with the plugin approach I'm suggesting?

[Phssthpok]

Originally Posted by eschwartz

Or simply add the magic directly to calibre via a source checkout.

Aaargh! No thanks! Then I'd have to re-insert the change and rebuild every time a new version comes out...

[Phssthpok]

Originally Posted by BetterRed

@Phssthpok - FWIW - if you put a shortcut to calibre.exe your Windows Send To folder then you can send a file with any of the common attributes set - readonly, system, hidden, archive. Works from a compressed volume, not sure about an encrypted volume.

You can send multiple 'books', if you want to pick and choose what to send, the check boxes in Windows file manager are useful. For me at least this is a 'better' option than Autoadd.

BR

Good idea -- but "compressed volume" limits me to zipfiles AFAIK. At the moment I use 7-zip, which seems to understand every format known to man.

[kovidgoyal]

Originally Posted by Phssthpok

All I really want to know at this point is, are there any foreseeable problems with the plugin approach I'm suggesting?

Like I said its way more complicated than it needs to be and you are messing with an already fragile process. But sure, in theory you could do it in a plugin. There is no hard boundary between core calibre code and plugins, so you can do pretty much anything in a plugin.

[BetterRed]

Originally Posted by Phssthpok

Good idea -- but "compressed volume" limits me to zipfiles AFAIK. At the moment I use 7-zip, which seems to understand every format known to man.

I don't where you got that idea, volume compression is a function of the file system. Folder/file compression is sometimes done by the operating system but more often by regular application programs.

My on-site backup disks are compressed volumes, on them I have zip, rar, 7z, sfx, iso, tar.gz, cbr, cbz, docx, epub etc etc. All of which I can uncompress with WinRAR (or Peazip or 7zip), to the same or another compressed volume or a regular non-compressed volume. And if I was minded I could install some plugins into my file manager and get it to do them.

BR