News feed - slashdot should be corrected or removed

[mike_bike_kite]

I just tried slashdot on the news feeds and my system seemed to be permanently gathering data. I looked more closely at the info and it stated that using this feed could get the users ip banned from slashdot. I went to slashdot and sure enough my ip is now banned!

Is there much point in keeping this feed if all it does is get the user banned? It appears there's a mobile page on slashdot (slashdot.org/palm) which might be a better place to grab the info but I couldn't test this as I'm banned

Mike

[kovidgoyal]

The operative word is "might" get you banned. It works for some people and not for others.

If you're willing to develop a better recipe, I'll be happy to replace that one.

[thinkpad]

How can they ban your IP everyone else with that IP will also be banned?

[Starson17]

Originally Posted by mike_bike_kite

I just tried slashdot on the news feeds and my system seemed to be permanently gathering data. I looked more closely at the info and it stated that using this feed could get the users ip banned from slashdot. I went to slashdot and sure enough my ip is now banned!

There are some recipe throttles available that may help. You can reduce the number of simultaneous connections and add a delay as follows:

simultaneous_downloads = 1
delay = 3

These may help in avoiding an IP ban. Your testing may help others when the ban is lifted (or your IP changes)

[mike_bike_kite]

Originally Posted by kovidgoyal

The operative word is "might" get you banned. It works for some people and not for others.

If you're willing to develop a better recipe, I'll be happy to replace that one.

Surely someone would only want to have slashdot on their ereader if they enjoy reading that site. If using the tool means a 50% chance of loosing all access to the site then it's probably best not having it there at all. My skill set is SQL, Unix shell (awk regexpr etc), PHP, html. If I'm useful and you have an example I can follow then I'm happy to help.

Originally Posted by thinkpadx

How can they ban your IP everyone else with that IP will also be banned?

Why would they care? If they have an ip that's killing their server then they'll just ban the ip. I've got my fingers crossed at the moment that they'll let me back on.

[Starson17]

Originally Posted by thinkpadx

How can they ban your IP everyone else with that IP will also be banned?

There won't be anyone else with that IP when the banned user is using it, and the ban is probably short-term so that by the time it is reassigned, the ban is lifted.

[mike_bike_kite]

Originally Posted by Starson17

There won't be anyone else with that IP when the banned user is using it, and the ban is probably short-term so that by the time it is reassigned, the ban is lifted.

I'm pretty sure all the computers in my house (wife, kids etc) have the same IP address. Also wouldn't a company use a NAT to show a single public ip address to the world while there might be 1000's of computers behind this address. I'll admit I'm a little unsure when it comes to networking though.

[thinkpad]

Originally Posted by mike_bike_kite

I'm pretty sure all the computers in my house (wife, kids etc) have the same IP address. Also wouldn't a company use a NAT to show a single public ip address to the world while there might be 1000's of computers behind this address. I'll admit I'm a little unsure when it comes to networking though.

That's my impression to. Slashdot risk banning thousand of IP addresses depending on how many computers is behind the NAT, risky business.

[Starson17]

Originally Posted by thinkpadx

That's my impression to. Slashdot risk banning thousand of IP addresses depending on how many computers is behind the NAT, risky business.

You're both right, but they seem to think banning is preferable.

[kovidgoyal]

They're in the happy position of having more readers than they need

[Starson17]

Originally Posted by Starson17

There are some recipe throttles available that may help. You can reduce the number of simultaneous connections and add a delay as follows:

simultaneous_downloads = 1
delay = 3

With these settings I was not banned. It takes forever, so faster settings may also work, but AFAICT, these seem to work to prevent banning.

[Marauder]

Originally Posted by kovidgoyal

If you're willing to develop a better recipe, I'll be happy to replace that one.

I'm game... : o )

Originally Posted by mike_bike_kite

Why would they care? If they have an ip that's killing their server then they'll just ban the ip. I've got my fingers crossed at the moment that they'll let me back on.

And that's exactly what they're doing. Keep in mind that Slashdot is an EXTREMELY high volume site. What it seems they do it if an IP exceeds a certain threshold of (requests/bandwidth over time?) they ban the IP in order to preserve load and resources for the rest of the world. Basically, Slashdot bans the IPs to avoid getting Slashdotted themselves...

Originally Posted by Starson17

There won't be anyone else with that IP when the banned user is using it, and the ban is probably short-term so that by the time it is reassigned, the ban is lifted.

That does not seem to be the case. The IP from my primary ISP was banned some time last year (it's been long enough that I forget exactly when) and despite numerous pleas to several different email addresses associated with Slashdot and OSDN (their parent company), I have received absolute silence.

Thanks to some static routes I set in my router/firewall, I am still able to access Slashdot by sending all traffic to their servers (two IP's responsible for the site) through a secondary/redundant IP/ISP. My primary IP remains banned...

(Now, if Slashdot would just start supporting IPv6 and publish AAAA records...)

Originally Posted by mike_bike_kite

I'm pretty sure all the computers in my house (wife, kids etc) have the same IP address. Also wouldn't a company use a NAT to show a single public ip address to the world while there might be 1000's of computers behind this address. I'll admit I'm a little unsure when it comes to networking though.

A home broadband router uses NAT to allow the multiple home computers to access the outside world through the single public IP assigned by the ISP.

From a business perspective, you typically won't have multiple thousands of IP's behind a single NAT-ed address. One of the biggest reasons is performance and other limitations of NAT. The NAT router needs to keep track of every active connection so that when it receives reply traffic from the outside world it knows the original internal host to send that traffic to. Large networks also have to be split into subnets. While several subnets can share a single NATed address, in a large enough installation, such as when you have thousands of users, there will be at the very least be a much larger pool of NAT addresses that are used, so instead of Many <-> One, it's a Many <-> Many translation which distributes the load better.

Anyway, if a large organization get's itself banned from it's employees browsing Slashdot, then there are probably more pressing matters (such as employee productivity) that need to be addressed. And in a sideways way, Slashdot would be doing them a favor by banning the IP. :-)

Originally Posted by thinkpadx

That's my impression to. Slashdot risk banning thousand of IP addresses depending on how many computers is behind the NAT, risky business.

But that's rarely the case that a single IP has many thousands of systems behind it. A network with thousands of NAT-ed hosts will typically have many public facing IPs for that network as well.

As a Network Engineer for a rather large, and high profile, Government Agency, I can understand blocking IPs that cause service degradation. (in essence it could be a DoS - Denial of Service - attack against Slashdot, of course Slashdot is a DDoS against every other site it links to). Not only do we regularly block IP's on both temporary, and permanent, basis when we see malicious traffic, there have been entire netblocks blocked from time to time when they've been the source of a high number of attacks...

[mike_bike_kite]

Thanks for the intel Marauder. If you do rewrite the slashdot feed would you just pull the top page with the descriptions and then each sub page with only the highly ranked comments. Could you then cache these results (perhaps for a couple of hours) so users don't have to reload everything from scratch each time. This should make the feed instant for nearly everyone and no one would get banned.

[Starson17]

Originally Posted by Marauder

I'm game... : o )

AFAICT, the throttles work - you just need to tweak them to deal with the banning issue. The results, however, included a lot of junk. While I read Slashdot, I would miss the ability to respond and to display limited subsets of commentary if I read it via the recipe/EPUB. If you are going to work on it, you probably ought to clean it up after you decide how to adjust the throttles.

That does not seem to be the case. The IP from my primary ISP was banned some time last year (it's been long enough that I forget exactly when) and despite numerous pleas to several different email addresses associated with Slashdot and OSDN (their parent company), I have received absolute silence.

Interesting. I've looked at each case of Slashdot banning I've seen reported here. I've tested several versions of the Slashdot recipe to decide if throttling helps. I've been banned once during those tests (unthrottled - one ban out of 5-6 unthrottled tests), but quickly changed my dynamic IP and tested some more. Someone out there must still be suffering from that old IP ban. You would think that most users to Slashdot are not on static IPs, so banning by IP for long periods wouldn't help much.

[Starson17]

Since this thread started, I've been testing Slashdot with these two throttles added to the recipe:

             simultaneous_downloads = 1
             delay = 3

It's worked reliably (but slow) every day, without banning me.