Wallabag (read it later app) automated synchronization

[anarcat]

Hi!

I have been working for the last few days on a program to synchronize a Kobo device with a Wallabag instance. Wallabag, if you do not know, is a free software replacement for Pocket, a "read it later" app.

The software is available on Gitlab, where you can also download actual binaries that you can install directly on your Kobo, including an (untested) KoboRoot.tgz that you should be able to just drop on your SDcard and start syncing! (Some configuration is actually required to write your credentials.) More information is available in the project's README file.

This targets Kobo readers, but it may just work on other e-readers, provided they have an architecture supported by Golang and you can find a way to notify the OS about the new books. I posted this in a Kobo topic just because it's the tool I use right now, but it may be better in a different topic...

Feedback, testing and comments are very welcome.

Note: this is a followup to a thread I posted about a year ago, it's nice to finally scratch that itch. Also note that I had trouble making the books synchronize, and started a discussion about the best way to trigger a library rescan, so your feedback is welcome there as well.

[Ken Maltby]

"chrox" was considering adding a Wallabag plugin for KOReader , I don't know what might have been done on it though. You might want to contact him through the project at Github.

Luck;
Ken

[anarcat]

Originally Posted by Ken Maltby

"chrox" was considering adding a Wallabag plugin for KOReader , I don't know what might have been done on it though. You might want to contact him through the project at Github.

Luck;
Ken

thanks Ken! i posted a comment in here and here.

[kido.resuri]

I may be interested in this Wallabag, if it could efficiently convert the webpages to epub. Pocket sadly fails to be good at this, at least on the webpages that my fiancée would like to see on the reader (mostly cooking recipes that we cook together from).
Pocket is unable to save them or only parts, and the ingredients or even the whole recipe is left out when saved to pocket.
If you have Wallabag installed and properly configured, may I send you some links to test?

[anarcat]

Originally Posted by kido.resuri

I may be interested in this Wallabag, if it could efficiently convert the webpages to epub. Pocket sadly fails to be good at this, at least on the webpages that my fiancée would like to see on the reader (mostly cooking recipes that we cook together from).
Pocket is unable to save them or only parts, and the ingredients or even the whole recipe is left out when saved to pocket.
If you have Wallabag installed and properly configured, may I send you some links to test?

Sure. you can also try out public instances:

• official service (14-day free trial)
• framasoft site (donation based)

[kido.resuri]

Please try these links:
http://www.mindmegette.hu/csicseribo...boccal.recept/
http://www.nosalty.hu/recept/friss-levegos-pita
http://azila.gportal.hu/gindex.php?pg=29932127
http://aprosef.hu/sult-oldalas-recept
http://vegalife.blog.hu/2015/04/01/h..._vagdalt_vegan
http://rupanerkonyha.cafeblog.hu/201...borekos-hutos/

Thanks!

[anarcat]

i won't generate an epub for all of those - i encourage you to register to the free services to try them. here's the first one as an epub: https://share.riseup.net/#swMetLQNjm0nh8WcmVwpqg

[kido.resuri]

Thanks. It did generate, quite well from the page. Pocket had trouble with this particular site, only had link, not the content. I'm not totally satisfied with it tho, but it may change. Thank You!

[oschene]

Works on the Mini. Slick!

[AmpersandNYC]

Has anyone got this working on an Aura One with the latest firmware? And is there a way to get it to put the epub files into a directory that koreader can read? I have a .books directory at the top level of my Aura One that I'd prefer the articles go into.

What I really want is a full Pocket or Instapaper client that I can launch from KSM. But this is 100% open-source so I think it's worth working on.

[anarcat]

you can customize where the files are written with the -output parameter on the commandline or (more likely) the "Output" setting in the wallabako.js file.

can't help you with instagram or pocket, unfortunately. only free software here. you may want to look at that issue for KSM support in wallabako, however: https://gitlab.com/anarcat/wallabako/issues/15

[bhyde]

I've been using this for a week now, with framabag.org. I can't thank you enough. Awesome. FYI - setting up the android app first helped to be sure I had all the right settings.

[jasio]

I've been experimenting a bit with wallabako and KOReader.

Wallabag is a bit of a pain to install but there is an experimental package for Synology NAS so I used that (I didn't check if it's a recent version but it does the job for now). There are also public servers, of course.

Since I'm always in KOReader, I don't need the tricks to update the Kobo database. I extracted the wallabag executable from the archive and copied it to /mnt/onboard.

I created a link to the CA file that already exist in KOReader with:

mkdir -p /etc/ssl/certs
ln -s /mnt/onboard/.adds/koreader/common/turbo/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

My config file with the server details is in /etc/wallabako.js but I might use the -config option to move it somewhere else.

I use this simple script to launch wallabako, /mnt/onboard/.adds/koreader/wb:

/mnt/onboard/wallabako -output "/mnt/onboard/Documents/Wallabag" -delete

Since there is no UI I run the script with the Terminal Emulator plugin, just typing:

ash wb

It does work pretty well. Sure, a koreader plugin with a UI to set it up and run it on demand would be nice.

But the thing really missing is the ability to delete documents from the server if they have been marked as Finished locally. That would require some work on wallabako itself though.

[anarcat]

Originally Posted by jasio

Since I'm always in KOReader, I don't need the tricks to update the Kobo database. I extracted the wallabag executable from the archive and copied it to /mnt/onboard.

Thanks for the review! I think it's the first report of Wallabako running on KOReader, so congratulations!

Originally Posted by jasio

I created a link to the CA file that already exist in KOReader with:

mkdir -p /etc/ssl/certs
ln -s /mnt/onboard/.adds/koreader/common/turbo/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

I should probably have documented this better, but the cryptographic libraries used by Wallabako Golang's crypto/x509 support reading other certificates databases:

On UNIX systems the environment variables SSL_CERT_FILE and SSL_CERT_DIR can be used to override the system default locations for the SSL certificate file and SSL certificate files directory, respectively.

So instead of creating a symlink, you could have simply changed the environment to have SSL_CERT_FILE point at ca-certificates.crt...

Originally Posted by jasio

My config file with the server details is in /etc/wallabako.js but I might use the -config option to move it somewhere else.

I use this simple script to launch wallabako, /mnt/onboard/.adds/koreader/wb:

/mnt/onboard/wallabako -output "/mnt/onboard/Documents/Wallabag" -delete

... for example, the above would become:

#!/bin/sh
SSL_CERT_FILE=/mnt/onboard/.adds/koreader/common/turbo/ca-certificates.crt
export SSL_CERT_FILE
/mnt/onboard/wallabako -output "/mnt/onboard/Documents/Wallabag" -delete

Originally Posted by jasio

Since there is no UI I run the script with the Terminal Emulator plugin, just typing:

ash wb

It does work pretty well. Sure, a koreader plugin with a UI to set it up and run it on demand would be nice.

Yeah, I've been meaning to do this forever, but never got around to actually doing it. It doesn't help that I'm still using Nickel: the integration with Wallabako is horrible, but I find the basic UI much better, I'm sorry to say. I should probably give KOReader a try again, but I would need to reinstall as it seems recent Kobo updates broke KSM...

Originally Posted by jasio

But the thing really missing is the ability to delete documents from the server if they have been marked as Finished locally. That would require some work on wallabako itself though.

I made the deliberate choice, in Wallabako, to treat the Kobo as "disposable" (ie. we can add/remove files from it, if the -delete option is set of course, as you probably noticed) and the Wallabag server as "precious" (ie. we *don't* delete files there). We *do* mark files as read on the server when they are read locally, but that's as much "writing" we do on the server side. So you could make a job that would periodically remove the read Wallabag articles from the server to fix your problem...

I keep all my articles in Wallabag, even when they're read. I find it's a great feature as I often go back and read stuff later. It does mean I have thousands of entries in there, but it still manages well, at least so far.

That being said, deleting articles on the server shouldn't be that hard, feel free to open a feature request on the topic in the issue tracker, but I don't guarantee I'll have time to actually implement it. Some other passer-by who wants to learn Golang (it's not hard for a novice programmer!) might just do that!

Thanks for the review, it's always nice to hear from users!

[jasio]

Thanks for the pointer to SSL_CERT_FILE. I have now changed my script to this (without the export step):

SSL_CERT_FILE=/mnt/onboard/.adds/koreader/common/turbo/ca-certificates.crt /mnt/onboard/wallabako -output "/mnt/onboard/Documents/Wallabag" -delete

This is much better since I was concerned that a software update would remove my link.

If I find the time, I'll try to do a minimalist koreader plugin to just run the script.

Nickel is pretty but KOReader is very powerful. I love the OPDS integration so I can get books from my own server or Feedbooks, ManyBooks... Now with a working Wallabag setup I'm super happy.

I'll open a request on the issue tracker for an option to delete files from the server. It would be a nice-to-have feature but I can live without.

Thank you for Wallabako!