Hacker News new | past | comments | ask | show | jobs | submit
vanyle | on: Residential proxies detected on White House network
Residential proxies are a very important part of the internet, they allow scraping for archiving, automation and can be used for extra anonymity.

While they have grown a bit too much, they are a natural response to the absurd amount of effort every website is pouring into anti-scraping tech.

10111001 | parent | next
> Residential proxies are a very important part of the internet, they allow scraping for archiving, automation and can be used for extra anonymity.

Aren't they almost always run unknowingly, by people tricked into running them?

dabbz | root | parent | next
There exist services that will pay you a fraction of money to host their residential proxy service on a computer you own. But I do think the majority of residential proxies are hijacked.
kube-system | root | parent
There are providers who claim they are ethically sourced, and that users have consented. Although I guess many are users who don't know or care that it is happening.

Ethically, if someone just wants to play a free game and doesn't care about it, is it okay? How knowledgable does a user have to be in order to have informed consent?

10111001 | root | parent
> There are providers who claim they are ethically sourced, and that users have consented. Although I guess many are users who don't know or care that it is happening.

Betcha that means there was "consent" language on page 55 of some click-through EULA.

kube-system | root | parent
The landscape spans the gamut of everything from malware to explicit consent: https://packetstream.io/share-bandwidth/
xyzzy_plugh | parent | next
Leveraging compromised devices is a very important part of the Internet??? This is an incredibly twisted perspective.

Help me understand. As far as I'm aware, these residential proxies are almost always malware.

lakeviewterrace | parent | next
"Malware is an important part of the Internet"

> and can be used for extra anonymity

Great, so you can conduct illegal transactions and have some unsuspecting grandma take the blame and have her front door kicked in?

kube-system | root | parent | next
Just like Tor, encryption, VPNs, etc, there are tons of legal uses for these proxies as well.
lakeviewterrace | root | parent
People who run Tor exit nodes generally do so by choice, and know the risks implications involved.

They didn't just connect a coffee pot to their WiFi one day.

kube-system | root | parent
Not all of these networks are built via malware.

e.g. https://packetstream.io/share-bandwidth/

timacles | root | parent
"Malware is an important part of the Internet

Technically correct statement though

OptionX | parent | next
And making DDoS-attacks nigh impossible to IP block.

Residential proxies are in many cases at least shady, both on the side of who uses them and for what and on providers that are often not above using dark pattern to hide them in some innocuous app for a smart-whatever that silently runs a proxy on a unaware user's connection. And in many places in the world if illegal stuff in done using your connection you are liable, regardless if you knew about it.

benjojo12 | parent | next
Did a residential proxy provider write this
lakeviewterrace | root | parent | next
Huh, and I wonder why I've been getting tons of "you might be a bot" CAPTCHA challenges from Google lately.
mcmcmc | root | parent
Frontier lab training data “collector”
georgel | parent | next
I wouldn't call turning on CloudFlare's bot blocker an "absurd amount of effort" from the perspective of site owners.

But getting around it sure is annoying, sometimes I get flagged IP's from Comcast and can't visit sites like costco.com despite not doing any questionable stuff on my network.

mcmcmc | parent | next
Hard disagree. They are malware 99.9% of the time. The utility does not erase the fact they are abusive.
mikestew | parent | next
Residential proxies are a very important part of the internet

So important that we have to trick people into using them. Remember when folks were running SETI@Home? Of course you do, because it was widely thought to be a good thing, and folks willingly signed up.

How many would buy the "box" if the advertising were honest about what's actually in the box?

Bender | parent | next
Most tools can be dual-use. Residential proxies are also used by malevolent actors to hack infrastructure or financial institutions and let some other poor soul get investigated for it. This is why we can't have nice things.
kube-system | root | parent
They're also used by the good guys who are going after those exact same bad guys.
Bender | root | parent
True, though unfortunately people can not be trusted to declare their intentions so the residential proxies will ultimately have to be sacrificed and given they are fairly easy to identify by behavior I could foresee the feds now having a reason to round 'em up faster. They already took out NetNut, IPIDEA, SocksEscort, Doppellganger and the "Kim Wolf" Offshoots.
kube-system | root | parent
Some of the more legit providers have KYC in place, and do literally require you to declare you intentions to open an account, and block certain traffic proactively unless you can prove a legitimate use.
Bender | root | parent
How do I identify which residential proxies have gone through KYC and which are wanting to pour hot sauce in my under garments?
kube-system | root | parent
The ones that do KYC list it pretty prominently on their website.
Bender | root | parent
They list the IP addresses of all the residential proxies? Is there a link to the netsets of all the IPv4 addresses they are using I can download once a day? (preferably over rsync but git over https is fine too)
amritananda | root | parent | next
I don't think even the most "ethical" providers are going to volunteer that information, since that defeats the entire purpose of having a residential proxy in the first place.
kube-system | root | parent
No...? That's a completely different question than what we were talking about above? The customers of a residential proxy who would be subject to KYC are the people using the addresses -- this has nothing to do with the exit addresses of those who have joined the network.
Bender | root | parent
That's a completely different question than what we were talking about above?

In fairness to me, I did say "Most tools can be dual-use. Residential proxies are also used by malevolent actors to hack infrastructure or financial institutions and let some other poor soul get investigated for it. This is why we can't have nice things."

So it could be Alice. I know Alice has good intentions. It could be Bob, we know what Bob is up to but I will give him a pass regardless because Bob did nice things in the past. But then there is Mallory ... Mallory Malfoid is up to no good. I need to know which IP's are used by Alice and Bob so I can exclude them from my automated protections. Mallory will automatically get rate limited to 16kb/s, 1 HTTP request every 5 seconds and if they bounce around different residential proxies they get redirected to a bathtub regardless of which residential proxy they use.

kube-system | root | parent
Yes, but KYC is a process where the service provider is making that determination. Either to a legal standard and/or their own standards.

In no world does "KYC" mean "expose your customer list to the whole world", let alone undermine the entire purpose of the product you're selling.

"I want to dox all of the customers of a proxy provider" is just an unreasonable ask. We wouldn't expect or want ISPs or VPNs or telecoms to do this.

loading story #50373016
mzajc | parent | next
s/Residential proxies/DDoS networks/, then re-read your comment.
sparkling | parent | next
>absurd amount of effort every website is pouring into anti-scraping tech.

Maybe some people do not want to provide their entire website as free training material to [insert LLM lab here]?

ethin | root | parent
You... Do know that people scrape websites for non-AI reasons, right?
anonreplier | root | parent
Why should the site owners be interested in allowing them to do that?
loading story #50378349
squidmaster | parent
Virtually all of it is the product of malware. This is like you saying that robbery is very important because you like buying cheap tools at pawnshops.