Ruling: IP address does not identify person

[JoeD]

Originally Posted by HarryT

Yea - if you have 100,000 lines the same, they will essentially be replaced by one marker saying "repeat this 100,000 times". If you made your file 10 million lines long, it would still compress to 8k . Artificial cases like that aren't a terribly good test, because they will compress in a way that "real" data doesn't.

Agreed.

For a client list to be realistic, every single line must be a unique email address. There will be repeated domains in the list where you're keeping a note of every employee/contact at a given client's company, but the names themselves are then going to vary.

The reason I said very unlikely rather than impossible, is you can have a 10k file expand to a gigabyte or more if the data is really heavily repetitive (whether it's useful data or not is a different matter :P). It's possible the file could be compressed highly if you only have a handful of different companies listed, giving a large repetition of domain names.

To test that though, you'd have to at the very least generate a file with realistic first/last names. You could allow the first or lastname to be repeated throughout a single or multiple companies, although, chances are you'd find a wide range of names for any one company especially last names. However the first.lastname pair itself would have to be unique for any single company.

If it's a client list of just 55 companies with 10,000 employees at each, there's going to be plenty of domain repeats, but would the names push it over? If it's a client list of more than 55 companies which would allow name pairs to repeat more, the domains then might push it over.

[ApK]

I found a legitimate 45MB log file, that I skimmed to make sure had typically varied data (as I have many huge debug logs filled with repetitive "I'm stuck on this one file" lines.)

This 45MB log compressed to under 2 MB with 7z compression, and 4MB with zip compression.

By comparison, an 80MB error log with different time stamps on every line but mostly the same message, went down to 470K.

Do remember that while "real" data, especially logs, is not super repetitive, neither is it super random.

OK, this was fun. Back to work for me now.

[ApK]

Originally Posted by JoeD

Agreed.

For a client list to be realistic, every single line must be a unique email address. There will be repeated domains in the list where you're keeping a note of every employee/contact at a given client's company, but the names themselves are then going to vary.

If I had to guess, without any more info to go on, I'd bet a "client list" of 540000 entries will have a huge number of comcast.net, hotmail.com, and gmail.com type domains all through it.
There is a relatively small set of companies who would have anywhere close to 500,000 distinct domains in their 'client' list...not counting mass mailers.

[JoeD]

Originally Posted by ApK

If I had to guess, without any more info to go on, I'd bet a "client list" of 540000 entries will have a huge number of comcast.net, hotmail.com, and gmail.com type domains all through it.
There is a relatively small set of companies who would have anywhere close to 500,000 distinct domains in their 'client' list...not counting mass mailers.

That's quite likely yes, partly that's why I was assuming so few companies in the list to give the largest chance of high repetition. Even so though, the pre @ part would need to be unique where the same domain occurs.

[MikeB1972]

Created a file with 2 million random "email addresses"
random 10 first characters all ending in @yahoo.com (So all short addys with same domain)
Text Size 41.9MB
7z Ultra Compression 12.2MB
Upping the random characters to 15
Text size 51.4MB
7z Ultra Compression 18.7MB

[wodin]

Never mind

[HarryT]

Originally Posted by MikeB1972

Upping the random characters to 15
Text size 51.4MB
7z Ultra Compression 18.7MB

... which is pretty much the level of compression I'd expect to see from "real" text - 2 to 3x.

[ApK]

Originally Posted by HarryT

... which is pretty much the level of compression I'd expect to see from "real" text - 2 to 3x.

I would have expected data intentionally generated to be random, like the first half of those email addresses, to be much less compressible then an actual book, but indeed, the 3.1MB text file of War and Peace from PG goes down to only a little under 1MB, so there you go.

ApK

[morantis]

Wow, as I was out of the office all day, it took a good twenty minutes to glance through all of the previous entries since my last. Sometimes I wonder why I am even bothering with this, but hey, whatever. As per one reply that quoted that I had not "proven" anything, that is fine, the fact is that I have nothing to prove, I made a statement, it is true, part of what we do involves grabbing those logs from a random router sometimes, and I don't even look to see whether logging is on or off because most web interfaces to commercial routers throw that log into an I-Frame and it is harder to read that way for us than to an SSH client. To more directly answer that one reply, I really do not care whether you believe it or not, it was a statement and it is true.

I noticed in another reply that someone had misread something that I wanted to clarify. My router does not run Apache, I was using the log file from one of our web servers as a comparison to the log file in my router. Just a point of clarification and nothing else.

As far as SSH, yes, most of the time we use Putty, personally I like the fact that if I am going to work with a specific router repeatedly and it is not in the same LAN, then I can save the settings and not have to look up or remember an IP address, the only thing I do not like about Putty is the fact that there is a scroll back buffer.

I saw someone ask how to SSH into a router that has dd-wrt on it. I have not touched that custom OS for a long time and only worked with it maybe twice, we hit too many types and brands of routers and it seemed pretty limited to what it could install on at the time I looked. That being said, I have no idea how you would get into it, but I was under the impression that the whole reason for installing dd-wrt was to open up more options and functionality for the router, so I assume that something simple like SSH is in there somewhere.

Our client list is from the past five years, updated weekly from five operating locations, six people each location. Using a simple search and replace, I was able to determine that around 30% were common email domains such as hotmail, Yahoo.....the rest are private email domains. Just as a point of trivia, it takes a separate computer, running nothing but the OS and our private mass mailing software a little over three days to send out a small email with minimal media. We sell a copy of it every once in awhile to a client who needs a little extra boost and have heard of it taking as long as five days, just a point of trivia.

Ports/ Varies by router and we maintain a file with them by brand and model. Rarely 22. Almost impossible to change. Most of them we acquired as we went along by ordering or requesting the technical handbook for each one we needed. Same thing for the default password and username. We shouldn't and wouldn't just hand that information out, mostly because people will just screw up their router. If you absolutely, must, to prevent the end of the world, and was told by the creator of the universe that you have to get into your router through that interface, then send me a private message on here or email me at terry@revolutionvps.net. I only show up here once a day, so don't be surprised if emailing me is the better way.

[Rob Lister]

Originally Posted by morantis

Wow, as I was out of the office all day, it took a good twenty minutes to glance through all of the previous entries since my last. Sometimes I wonder why I am even bothering with this, but hey, whatever. As per one reply that quoted that I had not "proven" anything, that is fine, the fact is that I have nothing to prove, I made a statement, it is true, part of what we do involves grabbing those logs from a random router sometimes, and I don't even look to see whether logging is on or off because most web interfaces to commercial routers throw that log into an I-Frame and it is harder to read that way for us than to an SSH client. To more directly answer that one reply, I really do not care whether you believe it or not, it was a statement and it is true.

I noticed in another reply that someone had misread something that I wanted to clarify. My router does not run Apache, I was using the log file from one of our web servers as a comparison to the log file in my router. Just a point of clarification and nothing else.

As far as SSH, yes, most of the time we use Putty, personally I like the fact that if I am going to work with a specific router repeatedly and it is not in the same LAN, then I can save the settings and not have to look up or remember an IP address, the only thing I do not like about Putty is the fact that there is a scroll back buffer.

I saw someone ask how to SSH into a router that has dd-wrt on it. I have not touched that custom OS for a long time and only worked with it maybe twice, we hit too many types and brands of routers and it seemed pretty limited to what it could install on at the time I looked. That being said, I have no idea how you would get into it, but I was under the impression that the whole reason for installing dd-wrt was to open up more options and functionality for the router, so I assume that something simple like SSH is in there somewhere.

Our client list is from the past five years, updated weekly from five operating locations, six people each location. Using a simple search and replace, I was able to determine that around 30% were common email domains such as hotmail, Yahoo.....the rest are private email domains. Just as a point of trivia, it takes a separate computer, running nothing but the OS and our private mass mailing software a little over three days to send out a small email with minimal media. We sell a copy of it every once in awhile to a client who needs a little extra boost and have heard of it taking as long as five days, just a point of trivia.

Ports/ Varies by router and we maintain a file with them by brand and model. Rarely 22. Almost impossible to change. Most of them we acquired as we went along by ordering or requesting the technical handbook for each one we needed. Same thing for the default password and username. We shouldn't and wouldn't just hand that information out, mostly because people will just screw up their router. If you absolutely, must, to prevent the end of the world, and was told by the creator of the universe that you have to get into your router through that interface, then send me a private message on here or email me at terry@revolutionvps.net. I only show up here once a day, so don't be surprised if emailing me is the better way.

So that seems to be a very long way of not addressing the original request. I'll repeat it here for the sake of those with short attention spans.



I don't want to say you're being evasive but nothing in the post above relates to the claim you made and my request for evidence of it. I'll make it simple for you:

You claimed, and I quote:

Just to let you know, having the logging function disabled on the router does not stop logs from being created, you simply don't have access to them through the consumer or administrator interface. I can go into any router anywhere anytime and pull every website visited, providing that it is not five years ago and the logs have gone past the allocated size, but I guarantee that I can pull at least 3 to 4 years worth of logs.

I have a Cisco E2000 router. I've had it for 2 years. So tell us what process or procedure you would use to pull logs for the last 2 years.

[keeska]

I'm not addressing the 5 year claim, just making the point that there is indeed information in a router that can be used to get information about the whole path (or, 'route') of traffic independent of any management logging function.

A router does not know what a "whole path" is let alone how to get one. A router maintains a table of the destination networks/hosts it needs and the next hop to get there. Beyond the next hop it is clueless.

For virtually all home routers the next hop to get anywhere other then the local network it is connected to is the ISP. The router has only 2 routes - one for the local network and a "default" route. The default router tells it where to send all traffic it has no other route for.

Even commercial/enterprise routers (including those used by ISPs) do not know the whole path.

There are some tools which can try to trace a path between two hosts on the internet but they often fail since many people have disabled that functionality on their routers.

[keeska]

I don't want to say you're being evasive but nothing in the post above relates to the claim you made and my request for evidence of it.

Agree 100%

I have a Cisco E2000 router. I've had it for 2 years. So tell us what process or procedure you would use to pull logs for the last 2 years.

I'll make it even easier for you. I have 4 routers - 2 Linksys (E3200 and E4200) and 2 Cisco (1811 and 1841). Tell me how you will get any logs from any of them.

[Ekaros]

Haven't read the thread but logging all traffic?

Take just the IP address, That is 4bytes, if you want to pair it with where it came from, it's 8 bytes. That could be considered the low bound of any time you load new website. How many sites you visit per day 20? 80 bytes, for a year that is ~29kbytes. Still real number is much much higher. I might need to run Wireshark for day to get some data on just how many SYN-packets(opening new connection) is made.

EDIT: Few more things, I didn't calculate things like time 4 bytes or 8 bytes(future proof to sufficient distance), and ports, 2 bytes. Also 20 is very low estimate. It's more of line in log for each time you open new tab in browser. And this is just for HTTP and considering each site doesn't have content located on other servers.
Logging certain specific things is cheap, like port-scans and other abnormal behavior. But logging visited websites quickly explodes to huge amount of data.

Also one thing which can explain old times in log-files is that clock isn't battery back-up and as such it resets when power is lost for certain period of time.


Finding out route is achieved by different means than routers. Usually with option of TTL and multiple packets in which case you often get answer for each router when packet was lost.

[ApK]

Originally Posted by keeska

A router does not know what a "whole path" is let alone how to get one.

A single router doesn't, but, 1, I was referring to the whole path that THAT router handled, that is, the destination private IP and port in the LAN, to identify a particular device, as well as the public IP in a typical NAT/PAT home setup, which was the point under discussion, and 2, not that it matters in this context (and not that this context matters all so much....) it's possible (likely I'd think) that the whole path can be put together from information stored in various device on the along the path. This is how net criminals get caught.

[JoeD]

@morantis are you certain you've not misread the file size of your client list. Can you confirm what compression is used on the file?

Also, is the web server you mentioned hardly ever used? To have 7 years worth of logs come to under 87KB seems highly unlikely even when compressed unless it's unused most days.

I'd also echo the above posters in asking what command it is you're running when you SSH into the router to access the routers logs (it would also be helpful if you could say what model the router is, then everyone could look info up themselves and cut out some back and forth questions)

I know you don't _have_ to prove anything, but you made a number of claims that just sound wrong to a number of other people on here. In the end it doesn't matter who is right or wrong, but without you providing the additional information asked for, there's no way for us to know if we're wrong or not.

Our client list is from the past five years, updated weekly from five operating locations, six people each location.

6 people at each of the five locations would be 30 email addresses per week added to the file. After 5 years that'd be under 10,000 email addresses. How come the file has half a million entries if that's the case? Also every week it is updated, are the new email addresses added unique or potentially repeats from previous weeks?

Both of those could have a huge impact on whether the file could make 37KB or not.