fuseflt - A FUSE filesystem with file conversion filter support

[twobob]

[root@kindle root]# /mnt/us/extensions/fuseflt/fuser --help

fuseflt 0.0.9 - A FUSE filesystem with file conversion filter support
Copyright (c) 2007 Theodoros V. Kalamatianos <nyb@users.sourceforge.net>

usage: /mnt/us/extensions/fuseflt/usr/bin/fuseflt configfile sourcedir mountpoint [options]

general options:
-o opt,[opt...] mount options
-h --help print help
-V --version print version

FUSE options:
-d -o debug enable debug output (implies -f)
-f foreground operation
-s disable multi-threaded operation

-o allow_other allow access to other users
-o allow_root allow access to root
-o nonempty allow mounts over non-empty file/dir
-o default_permissions enable permission checking by kernel
-o fsname=NAME set filesystem name
-o subtype=NAME set filesystem type
-o large_read issue large read requests (2.4 only)
-o max_read=N set maximum size of read requests

fuse: warning: library too old, some operations may not not work
-o hard_remove immediate removal (don't hide files)
-o use_ino let filesystem set inode numbers
-o readdir_ino try to fill in d_ino in readdir
-o direct_io use direct I/O
-o kernel_cache cache files in kernel
-o [no]auto_cache enable caching based on modification times
-o umask=M set file permissions (octal)
-o uid=N set file owner
-o gid=N set file group
-o entry_timeout=T cache timeout for names (1.0s)
-o negative_timeout=T cache timeout for deleted names (0.0s)
-o attr_timeout=T cache timeout for attributes (1.0s)
-o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout)
-o intr allow requests to be interrupted
-o intr_signal=NUM signal to send on interrupt (10)
-o modules=M1[:M2...] names of modules to push onto filesystem stack

-o max_write=N set maximum size of write requests
-o max_readahead=N set maximum readahead
-o async_read perform reads asynchronously (default)
-o sync_read perform reads synchronously

Module options:

[iconv]
-o from_code=CHARSET original encoding of file names (default: UTF-8)
-o to_code=CHARSET new encoding of the file names (default: UTF-8)

[subdir]
-o subdir=DIR prepend this directory to all paths (mandatory)
-o [no]rellinks transform absolute symlinks to relative
[root@kindle root]#

[twobob]

why is this?

There's a FUSE filesystem called fuseflt that does almost what you want. Rather than mounting a single file, it mounts a directory. Within the mount, you can set up arbitrary filters. The documentation gives your specific example: mounting a directory full of zipped files so that they appear to be unzipped.

Hopefully might have some value for you Silver18.

[twobob]

README: (just in case, these things disappear)

fuseflt - A FUSE filesystem with file conversion filter support

Contents:

1. Introduction
2. Platforms
3. Setup
3.1. Compilation
3.2. Installation
3.3. Configuration
3.3.1. Extension filter example
3.3.2. File conversion filter example
3.4. Running
4. Internals
5. License
6. Authors


1. Introduction

fuseflt is a FUSE filesystem that allows the user to define file conversion
filters that will be applied when requested. It relies on filename extensions
to determine file types.



2. Platforms

Currently only the following platforms are supported:

* Linux 2.6.x

It may work on other platforms with a FUSE port. You need the libcfg+ library,
which can be found at http://platon.sk/projects/libcfg+/.



3. Setup

3.1. Compilation

First of all you need the FUSE library and headers (http://fuse.sourceforge.net)
as well as the libcfg+ library (http://platon.sk/projects/libcfg+/). Afterwards,
a simple call to `make' is normally all you will need to compile fuseflt:

$ make

You can override the CFLAGS variable to provide optimisation flags e.t.c. The
DEBUG variable can be used to enable debugging - if you are using gcc, setting
DEBUG to "-g" would probably do.


3.2. Installation

First verify with `make -n install' the installation paths. By default, the
installation prefix is /usr/local and the fuseflt binary is installed in
$(prefix)/bin. You can set the prefix and bindir variables to override these
defaults. Then run `make install' as root to perform the actual installation:

# make install


3.3. Configuration

The configuration resides in a plain text file, which is parsed using the
libcfg+ library. It contains variables in the following form:

<variable> = <value>

The supported variables are:

cache_chk_int: an integer that indicates the number of seconds to elapse before
each cache entry expiry check. It can be specified only once.

cache_max_age: an integer that indicates the maximum allowed age of a cache
entry in seconds, after which the entry is removed. It can be specified
only once.

cache_max_nr: an integer that indicates the maximum number of cache entries.
Since each cache entry contains an open file descriptor, setting this
may be necessary to keep fuseflt from running out of file descriptors.
The cache tree will be pruned each time this limit is reached.

temp_dir: the directory where the temporary files will be created.

ghost_temp: By default fuseflt uses "ghost" temporary files, files that have
been created and then unlinked. While this method has the advantage of
not leaving files around, it does require an open file descriptor for
each temporary file. By setting ghost_temp to 0, the total usage of file
descriptors is much saner, but the temporary files become visible and if
fuseflt is killed abruptly (kill -9, segmentation faults, the end of the
universe as we know it e.t.c.) they will have to be removed manually.

ext_in: along with a corresponding ext_out variable, it creates a filename
filter, where filenames that end in the string supplied in ext_in will
be changed to end in the one supplied in ext_out instead. Both variables
are strings and can be left empty if needed. Each ext_in should have
_exactly_ one corresponding ext_out entry. Multiple ext_in/ext_out sets
can be used to specify multiple filename filters.

ext_out: see ext_in above.

flt_cmd: a filter shell command. It is parsed using /bin/sh -c, so mind your
quotation.
flt_in: the input file extension for this filter.
flt_out: the output file extension for this filter.

All three flt_* entries are required to set a file conversion filter. Multiple
flt_* sets can be used to create multiple filters.


3.3.1. Extension filter example

Suppose that you already have a filter that decompresses .gz files, and you want
to use it transparently. All you have to do is add the following to your fuseflt
configuration:

ext_in = .gz
ext_out =

Each *.gz file will have the .gz suffix removed and then the empty string added,
essentially stripping the .gz extension.


3.3.2. File conversion filter example

Suppose that you want to convert all bzip2-compressed files to gzip-compressed
ones. The following should do:

flt_in = .bz2
flt_out = .gz
flt_cmd = bzip2 -dc | gzip -9c

A usage example:

$ ls
a.bz2 b.bz2
$ zcat a.gz
A A A A A
$ zcat b.gz
B B B B B

You may add the following in your configuration:

ext_in = .bz2
ext_out = .gz

to be more transparent about the whole process:

$ ls
a.gz b.gz


3.4. Running

Please read the FUSE documentation before using fuseflt.

Run `fuseflt -h' to see the various command line options. You need a prepared
configuration file beforehand and an existing mountpoint. Then all you have to
do in most cases is:

$ fuseflt file.conf source_directory/ mountpoint/

If you want to use fuseflt from /etc/fstab, a different syntax can be used. The
following example shows its use:

fuseflt##/etc/fuseflt.doc.conf##/usr/share/doc /fs/vfs/doc fuse defaults 0 0

To allow this, fuseflt supports an alternative command line syntax:

$ fuseflt '#file.conf##source_directory/' mountpoint/

If you do use fuseflt in your /etc/fstab, you will probably want to add the
allow_other and default_permissions options. The first option allows non-root
users to access fuseflt, while the second one activates the in-kernel permission
checking, since fuseflt does no access control on its own.

WARNING: _NEVER_ USE allow_other WITHOUT default_permissions, OR YOU RISK
ALLOWING UNPRIVILEDGED PROCESSES TO ACCESS FILES THEY SHOULD NOT BE ABLE TO.

Should you need to discard the file cache (e.g. when the source filesystem has
been altered, or if you want to free the open file descriptors), you can send
the USR1 signal to fuseflt to do so.



4. Internals

Some people say the most accurate documentation is the source code itself, no?



5. License

fuseflt is released under the GNU General Public License version 2. The full
text of the license can be found in the COPYING file that should be included in
the fuseflt distribution.



6. Authors

Original author:
Theodoros V. Kalamatianos <nyb@users.sourceforge.net>

[twobob]

@silver18
and - for the record - getting the supporting lib was a huge PITA... so I hope you do at least try it :P

[silver18]

I'm feeling important!
I really appreciate your work, I can't thank you the way you deserve.
I'll play with this in the weekend as I'm pretty busy at the moment.
By the way, I would like to mount cbz files (after renaming them to zip) to avoid the decompression.
It seems I need to study and test...

[hawhill]

If it's for that, I have made good experiences with https://github.com/hanwen/go-fuse and its zipfs example. It's written in Go which is a nice language and has a good self-contained way to cross-compile for ARM - an easy way to get code running on Kindles.

[silver18]

I didn't forget about this, I'm just a bit busy ATM!!

[silver18]

Originally Posted by silver18

I didn't forget about this, I'm just a bit busy ATM!!

Sorry again...I just don't want you to think I forgot this!

[twobob]

meh its all good. maybe I could build go-fuse at some point too...