COPS : Calibre OPDS PHP Server

[mariosipad]

Comparing different versions of the OS (Mint/Debian) and apache (2.2/2.4) I found that file names, content (what is in which file), order of statements in files, etc. can an will change.

However basically the functionality more or less remains the same. It's sometimes just confusing when comparing files from different PC's each with their own version of the OS and web server.

Regarding your "problem": could your installation of OwnCloud have something to do with it?
However it does not really matter to which log file Apache writes. Just use what Apache serves you.

I'm convinced your approach will work for you.
Maybe just search for "/cops/ " (space is intentional or "/cops/$)" ) instead of "getJSON".

[teejcee]

@mariosipad

I'll keep looking again today when time permits.
'Won't have a lot of time as I'll have my grandson here & he'll be wanting attention...as they do.

Cheers,
TC

[teejcee]

@mariosipad

I've finished my script, again with help from my mate, Garry.

#!/bin/bash
# email log entries of COPS users for previous 24 hours

DATE=$(date +%d-%b-%Y)
LOGFILE=/var/log/apache2/other_vhosts_access.log

cd ~/bin
# get logins  200 2183 are username/password logins
#             200 2328 are 'cached' connections

grep -e "200 2183" -e "200 2328" $LOGFILE > cops-log.txt

#   Get fields from cops_log to show user logins.

LOGFILE=cops-log.txt
TMPFILE=tmp.$LOGFILE

rm -f $TMPFILE 2> /dev/null

cat $LOGFILE 2> /dev/null | while read LINE
do
  IP=$(echo $LINE | cut -f2 -d' ')
  USER=$(echo $LINE | cut -f4 -d' ')
  DATETIME=$(echo $LINE | cut -f2 -d'[' | cut -f1 -d']')
  DATE=$(echo $DATETIME | cut -f1 -d'+')
  echo "$DATE $USER $IP" >> $TMPFILE
done

if [[ ! -s $TMPFILE ]]
then
  echo "There is no data to report." >> $TMPFILE
fi

cat $TMPFILE | mail -s "Today's COPS users $DATE"  me@myemail-addy

rm -f $TMPFILE 2> /dev/null

Once it's been running a while & I'm happy with it, I'll write temp files
out to /tmp

The grep command pulls entries with http status code 200 & return bytes
of either 2183 or 2328. I have no idea what these byte sizes are made up of, only as I've commented in the code.

Any questions...fire away.

Cheers,
TC

[mariosipad]

@teejcee

Hmm, I never thought that I would learn something about scripting by using COPS

I copied and installed your script into a debian/apache setup.

Out of the box it did not work so I "debugged" a bit.

Herewith my preliminary findings:

0 - I changed other_vhosts_access.log to access.log

1 - Using the line grep -e "200 2183" -e "200 2328" $LOGFILE > cops-log.txt didn't work for me.
a - neither 2183 nor 2328 showed up in my log file. But this was easily fixed with other (for me) matching numbers.
b - Logging into COPS with OPDS gave other numbers than logging into COPS from a browser. Did you take this into account in your script?
c - are you sure that one of these numbers isn't our old friend getJSON and double login log entries?

2 - Writing to TMPFILE did not get the desired result. I "blame" this on:
a - using another log file with a slightly different layout
b - Debian vs Mint date/time formatting (maybe?)
However even this was fixable with a lithe googling. Hey, I even learned something today!
How is the output to look like in your case? Just so I can compare.

3 - I did not (yet) use the mail functionality.

Greets

Mario

[teejcee]

Originally Posted by mariosipad

@teejcee

Hmm, I never thought that I would learn something about scripting by using COPS

Until retirement, I'd been in the computer game since the early 70's and
it was a dud day if I didn't learn something new. ( and still learning) .

0 - I changed other_vhosts_access.log to access.log

1 - Using the line grep -e "200 2183" -e "200 2328" $LOGFILE > cops-log.txt didn't work for me.
a - neither 2183 nor 2328 showed up in my log file. But this was easily fixed with other (for me) matching numbers.

This would be because the LogFormat defined in apache2.conf for
vhosts would be different . I believe if you followed %>s with %0 in
your log definition, you'll get that field. If that doesn't work, try %b

b - Logging into COPS with OPDS gave other numbers than logging into COPS from a browser. Did you take this into account in your script?
c - are you sure that one of these numbers isn't our old friend getJSON and double login log entries?

Hmmm, didn't consider OPDS....then again, I didn't tell my users about
it so it shouldn't matter. I'll try some testing & let you know.
I'm as sure as I can be about the codes. I did quite a bit of testing and viewed the original log file to work it out.
Mind you, I'd be a lot happier if I knew what those
byte counts represented.

Here is an edited sample output...

01/Feb/2016:11:17:04 aaaaa 110.175.xx.xxx
01/Feb/2016:11:20:41 aaaaa 110.175.xx.xxx
01/Feb/2016:11:24:08 aaaaa 110.175.xx.xxx
01/Feb/2016:11:25:48 aaaaa 110.175.xx.xxx
01/Feb/2016:15:32:04 bbbbb 192.168.1.1
02/Feb/2016:12:21:11 ccccc 58.173.aa.aaa
02/Feb/2016:12:55:58 ccccc 58.173.aa.aaa
02/Feb/2016:19:04:41 ccccc 58.173.aa.aaa
03/Feb/2016:11:46:24 bbbbb 192.168.1.1
04/Feb/2016:11:51:45 bbbbb 192.168.1.1
04/Feb/2016:14:40:00 bbbbb 192.168.1.1
04/Feb/2016:14:41:08 bbbbb 192.168.1.1
04/Feb/2016:15:30:39 bbbbb 192.168.1.1
04/Feb/2016:15:31:57 bbbbb 192.168.1.1
04/Feb/2016:15:34:27 bbbbb 192.168.1.1
04/Feb/2016:15:40:56 bbbbb 192.168.1.1

I'll probably add the "200 2183" etc codes eventually.

edit.... I've tried to login with opds & couldn't.
I used url/feed.php . Is that correct?

Cheers,
TC

[mariosipad]

OPDS = http://IP/cops/feed.php
[if you use cops in /var/www/html/cops ]
I'll get back on your other points tomorrow (as it's after midnight over here).

Greets,
Mario

[teejcee]

@mariosipad

I've got opds working...thanks for the tip.

In my testing, it looks like a successful login to opds returns "200 1035". It's been
consistent with every test.

Cheers,
TC

[mariosipad]

Mind you, I'd be a lot happier if I knew what those byte counts represented.

From the Apache documentation:
%b: Size of response in bytes, excluding HTTP headers. In CLF format, i.e. a '-' rather than a 0 when no bytes are sent.

Yeah thad did help

I experimented a bit with the %b values and found that different servers and different browsers and different url's all gave mostly different but not always different values of %b.
With my setup I got (at least) the following values of %b: (904, 906, 1057, 1058, 1061, 1062).
- the first 2 are with OPDS: http://cops/feed/php
- 1057 and 1061 are with url: http://cops
- 1058 and 1062 are with url's: httt://cops/ and http://cops/index.php

But your numbers differ from mine. Maybe %b is not a general solution to find unique lines in a log file?

Instead of %b I used the line

grep -e "/cops/ " -e "/cops/index.php " -e "/cops/feed.php " $LOGFILE | grep -e "200" > cops-log.txt

to find unique (succesful) logins.

However if all your "users" use the same browser and url

grep -e "200 1176" -e "200 1994" $LOGFILE > cops-log.txt

or whatever value (200 2183 ?) will work for you.


Just one question. Did you have to install anything to make mailing from your script possible or did it just work?


Greets,
Mario

[teejcee]

Originally Posted by mariosipad

Just one question. Did you have to install anything to make mailing from your script possible or did it just work?

Didn't install a thing. It just worked first up. Are you having troubles?

I'll look into the rest of your post later. Out of curiosity, where are
you located?

Cheers,
TC

[mariosipad]

When using the "mail" line in your script I get the error: "mail: command not found", which is to be expected because mail (or mailutils or any other mail program) is not installed.

And even if it was installed, but not configured properly, it would not work because my ISP does not allow outgoing mail on port 25 with my own SMTP program.

I need to use their SMTP server or use (for example) Google's.

I'm debating if it's worth the effort to install and configure (!) mail on my linux server(s). I do all my mail from either my desktop or one of my iOS (iPhone/iPad) devices.

To satisfy your curiosity. I'm located in the Netherlands, originally from Rotterdam, but living since the early seventies in the southwest part of the country near Breda.

[teejcee]

I need to use their SMTP server or use (for example) Google's.

You should be able to use sendmail in place of mail. I haven't of course but there are lots of how-to's on using sendmail to email to gmail.

I use google apps which is in essence gmail, as an IMAP server, and Thunderbird as the client, so if I get a chance today, I'll see if I can get it working.

Aside from that, there's also code in COPS that uses sendmail. If you're PHP
proficient, you may be able to use some of it as a basis. I know it works as
I have coded my details into config_local.php successfully.

I'm not sure of the difficulty or otherwise of setting up mail. It was installed as a default with Mint.

Googled Breda....looks nice.

Cheers,
TC

[mariosipad]

Originally Posted by teejcee

I use google apps which is in essence gmail, as an IMAP server, and Thunderbird as the client, so if I get a chance today, I'll see if I can get it working.

That would be nice. Hey, another opportunity to learn something!

Aside from that, there's also code in COPS that uses sendmail. If you're PHP proficient, you may be able to use some of it as a basis. I know it works as I have coded my details into config_local.php successfully.

I have used that, and it worked. But I know next to nothing of PHP.
And the challenge is now to make mail work from the command line or bash script. Not that I need it, but it's an opportunity to learn something new (every day)

It was installed as a default with Mint.

Well, it was not automatically installed on any of my devices either running Mint Cinnamon 17.0 nor Debian (Squeeze, Wheezy, Jessie) XFCE.

Googled Breda....looks nice.

Nothing wrong with your location on the other side of the world either

Mario

[teejcee]

Originally Posted by mariosipad

That would be nice. Hey, another opportunity to learn something!

Yep...

Here are a couple of links and a short explaination.

1. http://tecadmin.net/send-email-smtp-...nd-line-ssmtp/

2. http://unix.stackexchange.com/questi...al-smtp-server

It appears that mail & ssmtp can work hand-in-hand.

I used the first link to install and set up ssmtp, then discovered the second link.

In it, there's advice to install ssmtp & then mailutils. I was not aware util I read this that mail was part of mailutils. I had the instals the other way round so I remove both & re-installed in the order advised.

I have not needed to change a thing with my script....it still works.
However, in my case it worked anyway. So , follow the links instructions &
see if your mail gets past your ISP. Good luck.

Cheers,
TC

[mariosipad]

Using SSMTP and configuring it like COPS:config_local.php lets me mail OK!
I'm not 100% satisfied with how (it mail as root [in the mail from: field] and I do not like that, but changing this via "revaliases" does not seem to work.

Further I am looking into the packages mailutils and heirloom-mailx and figuring out which one is best to install.

Thank you for your links, they put me on the right track.

Edit: problems with heirloom-mailx (for now). Solving them is a nice task for tomorrow. Never mind, FIXED (typo )
And fixed my first "problem" with SSMTP as well for the most part. I can live with this.

[teejcee]

Originally Posted by mariosipad

And fixed my first "problem" with SSMTP as well for the most part. I can live with this.

So, I take it you can use mail with ssmtp to get past your ISP?

If so, "Tot de volgende keer"..... , thanks for all your input.

Cheers
TC