polipo - offline web proxy

[twobob]

[root@kindle bin]# /mnt/us/extensions/polipo/bin/polipo -v

configFile (none) Configuration file.
CHUNK_SIZE 4096 Unit of chunk memory allocation.
allowUnalignedRangeRequests boolean false Allow unaligned range requests (unreliable).
allowedClients list (not set) Networks from which clients are allowed to connect.
allowedPorts intlist 80-100, 1024-65535 Ports to which connections are allowed.
alwaysAddNoTransform boolean false If true, add a no-transform directive to all requests.
authCredentials atom (hidden) username:password.
authRealm atom (none) Authentication realm.
bigBufferSize integer 32768 Size of big buffers (max size of headers).
cacheIsShared boolean true If false, ignore s-maxage and private.
censorReferer tristate false Censor referer headers.
censoredHeaders list (empty list) Headers to censor.
chunkCriticalMark integer 24772608 Critical mark for chunk memory (0 = auto).
chunkHighMark integer 25165824 High mark for chunk memory.
chunkLowMark integer 18874368 Low mark for chunk memory (0 = auto).
clientTimeout time 2m Client-side timeout.
daemonise boolean false Run as a daemon
disableConfiguration boolean false Disable reconfiguring Polipo at runtime.
disableIndexing boolean true Disable indexing of the local cache.
disableLocalInterface boolean false Disable the local configuration pages.
disableProxy boolean false Whether to be a web server only.
disableServersList boolean true Disable the list of known servers.
disableVia boolean true Don't use Via headers.
diskCacheDirectoryPermissions integer 0700 Access rights for new directories.
diskCacheFilePermissions integer 0600 Access rights for new cache files.
diskCacheRoot atom /var/cache/polipo/ Root of the disk cache.
diskCacheTruncateSize integer 1048576 Size to which on-disk objects are truncated.
diskCacheTruncateTime time 4d12h Time after which on-disk objects are truncated.
diskCacheUnlinkTime time 32d Time after which on-disk objects are removed.
diskCacheWriteoutOnClose integer 65536 Number of bytes to write out eagerly.
displayName atom Polipo Server name displayed on error pages.
dnsGethostbynameTtl time 20m TTL for gethostbyname addresses.
dnsMaxTimeout time 1m Max timeout for DNS queries.
dnsNameServer atom 192.168.1.254 The name server to use.
dnsNegativeTtl time 2m TTL for negative DNS replies with no TTL.
dnsQueryIPv6 4-state false Query for IPv6 addresses.
dnsUseGethostbyname 4-state reluctantly Use the system resolver.
dontCacheCookies boolean false Work around cachable cookies.
dontCacheRedirects boolean false If true, don't cache redirects.
dontTrustVaryETag tristate maybe Whether to trust the ETag when there's Vary.
expectContinue tristate maybe Send Expect-Continue to servers.
forbiddenFile atom (none) File specifying forbidden URLs.
forbiddenRedirectCode integer 302 Redirect code, 301 or 302.
forbiddenTunnelsFile atom (none) File specifying forbidden tunnels.
forbiddenUrl atom (none) URL to which forbidden requests should be redirected.
idleTime time 20s Time to remain idle before writing out.
laxHttpParser boolean true Ignore unknown HTTP headers.
localDocumentRoot atom /mnt/us/extensions/system/etc/polipo/www/ Root of the local tree.
logFacility atom user Syslog facility to use.
logFile atom (none) Log file (stderr if empty and logSyslog is unset, /var/log/polipo if empty and daemonise is true).
logFilePermissions integer 0640 Access rights of the logFile.
logLevel integer 0x7 Logging level (max = 0xFF).
logSyslog boolean false Log to syslog.
maxAge time 14d1h Max age for objects without Expires header.
maxAgeFraction float 0.100000 Fresh fraction of modification time.
maxConnectionAge time 21m Maximum age of a server-side connection.
maxConnectionRequests integer 400 Maximum number of requests on a server-side connection.
maxDiskCacheEntrySize integer -1 Maximum size of objects cached on disk.
maxDiskEntries integer 32 File descriptors used by the on-disk cache.
maxExpiresAge time 30d1h Max age for objects with Expires header.
maxNoModifiedAge time 23m Max age for objects without Last-modified.
maxObjectsWhenIdle integer 32 Number of objects to write at a time when idle.
maxPipelineTrain integer 10 Maximum number of requests pipelined at a time.
maxSideBuffering integer 1500 Maximum buffering for PUT and POST requests.
maxWriteoutWhenIdle integer 65536 Amount of data to write at a time when idle.
mindlesslyCacheVary boolean false If true, mindlessly cache negotiated objects.
objectHashTableSize integer 32768 Size of the object hash table (0 = auto).
objectHighMark integer 2048 High object count mark.
parentAuthCredentials atom (hidden) username:password.
parentProxy atom (none) Parent proxy (host:port).
pidFile atom (none) File with pid of running daemon.
pipelineAdditionalRequests tristate maybe Pipeline requests on an active connection.
pmmFirstSize integer 0 The size of the first PMM chunk.
pmmSize integer 0 The size of a PMM chunk.
preciseExpiry boolean false Whether to consider all files for purging.
proxyAddress atom 127.0.0.1 The IP address on which the proxy listens.
proxyName atom kindle The name by which the proxy is known.
proxyOffline boolean false Avoid contacting remote servers.
proxyPort integer 8123 The TCP port on which the proxy listens.
publicObjectLowMark integer 1024 Low object count mark (0 = auto).
redirector atom (none) Squid-style redirector.
redirectorRedirectCode integer 302 Redirect code to use with redirector.
relaxTransparency tristate false Avoid contacting remote servers.
replyUnpipelineSize integer 1048576 Size for a pipeline break.
replyUnpipelineTime time 20s Estimated time for a pipeline break.
scrubLogs boolean false If true, don't include URLs in logs.
serverExpireTime time 1d Time during which server data is valid.
serverIdleTimeout time 45s Server-side idle timeout.
serverMaxSlots integer 8 Maximum number of connections per broken server.
serverSlots integer 2 Maximum number of connections per server.
serverSlots1 integer 4 Maximum number of connections per HTTP/1.0 server.
serverTimeout time 1m30s Server-side timeout.
smallRequestTime time 10s Estimated time for a small request.
socksParentProxy atom (none) SOCKS parent proxy (host:port)
socksProxyType atom socks5 One of socks4a or socks5
socksUserName atom (empty) SOCKS4a user name
tunnelAllowedPorts intlist 22, 80, 109-110, 143, 443, 873, 993, 995, 2401, 5222-5223, 9418 Ports to which tunnelled connections are allowed.
uncachableFile atom (none) File specifying uncachable URLs.

working on making this plug into skipstone initially

[root@kindle bin]# /mnt/us/extensions/polipo/bin/polipo -h

/mnt/us/extensions/polipo/bin/polipo [ -h ] [ -v ] [ -x ] [ -c filename ] [ -- ] [ var=val... ]
  -h: display this message.
  -v: display the list of configuration variables.
  -x: perform expiry on the disk cache.
  -c: specify the configuration file to use.

build for all kindles. TBC... Working on it. Got skipstone to play now...

man: http://www.pps.univ-paris-diderot.fr...po/polipo.html

[h1ro]

now make my 2TB hdd fit into that thing.

[twobob]

Hello? Networking protocols? Mount it as a virtual network drive or something :P

hehehe

[brianinmaine]

very handy for me! i wasn't even thinking about this - thanks!

[knc1]

Originally Posted by h1ro

now make my 2TB hdd fit into that thing.

Try (first) using the NBD (Network Block Device) driver.
We also have CIFS and NFS drivers for all kindles.

You might also want to set aside a partition of that 2Tbyte disk drive as a swap partition. (You can run swap on top of the NBD driver. GM was doing that over a year ago.)

[twobob]

I'm still struggling with the details of configuration, but it appears to work perfectly... If only I could figure out how to set it up!

[twobob]

nbd... these things may help...

[brianinmaine]

this is just some added info on the nbd stuff:
My (Debian laptop) Server:
# apt-get install nbd-server

create a 128M file to be made into a filesystem later on
# dd if=/dev/zero of=/home/user/nbd_file bs=1024 count=128000

The default port 10809 will be used.

/etc/nbd-server/config:
[generic]
[export]
exportname = /home/user/nbd_file

# mke2fs /home/user/nbd_file
(answer yes it is to be a special block device)
# chmod 766 /home/user/nbd_file <-nbd-server runs as nbd user? whatever..
# nbd-server &
optional (make sure it's listening)
# netstat -ltn | grep 10809



Kindle (client):

I put the nbd.ko driver in /mnt/us and nbd-client in my PATH

[root@kindle us]# insmod nbd.ko
[root@kindle us]# nbd-client 192.168.15.201 /dev/nbd0 -name export
[root@kindle us]# mkdir nbd
[root@kindle us]# mount /dev/nbd0 nbd
[root@kindle us]# ls -al nbd

places I ripped info from:
https://forums.gentoo.org/viewtopic-...eecfdaa988ceef
http://www.thegeekstuff.com/2009/02/...mpstart-guide/

[twobob]

Check the skipstone thread for the info I posted at the end on how to get that accepting proxies. Very simple, just in the .config file.

Nice work btw!

[brianinmaine]

Originally Posted by twobob

Check the skipstone thread for the info I posted at the end on how to get that accepting proxies. Very simple, just in the .config file.

ok, i give up. WHICH/WHERE is the .config file? Please?!

Nevermind:
/mnt/us/extensions/skipstone/settings/.skipstone/config

looks right to me! Thanks!

[twobob]

Hi, yes sorry! that one.

I saw you blink into the chat for a microsecond. Hope you are good mate.

: )

[knc1]

Originally Posted by brianinmaine

this is just some added info on the nbd stuff:
My (Debian laptop) Server:
# apt-get install nbd-server

create a 128M file to be made into a filesystem later on
# dd if=/dev/zero of=/home/user/nbd_file bs=1024 count=128000
- - - - -

You only need the first 16 sectors and the last 16 sectors zeroed.
The ext2/3/4 handles the hole gracefully.

You can get by with writing even smaller first and last "tracks" if not using LVM2 and/or encrypted disks.

PS: Thanks for the post. It should answer the question asked earlier of how a person could mount their 2Tbyte drive on the Kindle.

PS2: Note: swap will run on top of the NBD driver - set up a remote swap device/partition/file (any type in other words) for the Kindle to use as swap and then enable swapping on the Kindle.
Get rid of a lot of those "out of memory" problems.

[brianinmaine]

i did do a swap file also - you can use a few nbd devices at the same time this way, /dev/nbdX seems pretty neat. Still can't get polipo working right, though. seems a waste to me right now.