I have a fantastic HOSTS file (where can I post it for others tobenefit)?

On Sat, 30 Aug 2014 09:46:58 -0400, Big Al wrote:

How does one go about finding out all these bogus host
addresses so you can build your own HOST file. Or even, other than
the one link I did see, are there starter lists available?

There are many ways.

I started by searching the net, years ago, for all the hosts file
that were out there.

To eliminate duplicates, I removed all extraneous characters and sorted
the files, because nobody wants to go through this stuff by hand.

Here, for example, is just the one alias I use to strip extraneous
characters out of the MVP Hosts file:
alias cleanhost='cat HOSTS | grep -v 127.0.0.1 | grep -v "::" | sed -e "s/^M//g" -e "s/[ ]/ /g" -e "s/ / /g" -e "s/#.*$//" | sed -e "s/ $//" | sed -e "s/0.0.0.0/127.0.0.1/" | sort -u hosts.txt'

Anyway, then, EVERY time I have an obnoxious popup window, I put
the URL of that obnoxious window into the hosts file.

I use an ~/.exrc setting to make that easy to edit in the vi editor:
:map v :s;http://\([^/][^/]*\)\(/.*\)*;127.0.0.1 \1

I got that setting from this newsgroup actually, when I had searched
how to do it (I love this newsgroup because people think like I do).

Whenever an advertisement shows in the side, or those silly chat
windows, or a video that pops up on the side, etc., I just inserted
the domain of each of those obnoxious entities into the hosts file.

It used to be I was adding a dozen or score a day, and now it's
down to only one or three a week, since I have most of the bad
ones caught by now. The process will never end, but, we can pool
our efforts, which was why I posted the file in the first place.

On Sat, 30 Aug 2014 11:11:33 -0300, Shadow wrote:

You will have to close the browser and flush your DNS cache
for the changes to take effect.

This is an interesting point, as this has bugged me for years.

I found that Firefox just needs to be closed in order to flush
its cache of hosts. Or, you have to wait a rather long time,
even though you can set the cache flushing in about:config
(I've tried everything and have given up long ago).

You'd think the OS would flush DNS more often, but Firefox is
a weird beast when it comes to remembering the DNS cache.

On Sat, 30 Aug 2014 17:04:44 +0200, J.O. Aho wrote:

You also make your internet slower, not just for browsing, but for all
other activity too.

How does NOT visiting a web site, especially since these are all
obnoxious web sites, make the browsing slower?

On Sat, 30 Aug 2014 17:04:44 +0200, J.O. Aho wrote:

If you want to block sites, then use proper tools like privoxy or those
add blocker plug-ins, as this way you can get rid of the advertisement
without blocking legitimate content and you aren't slowing down for
other protocols.

I knew this would come up, and you're the first, I think, to bring it
up, but this thread isn't about the many (very many) other methods of
blocking unwanted domains.

On Sat, 30 Aug 2014 15:56:19 +0200, J.O. Aho wrote:

the hosts file was never intended for this amount of hosts and this
makes things slower...

On Linux, it does not seem to have any deleterious effect on speed.

In fact, one could argue the machine is *faster* since it doesn't
waste time visiting any number of a hosts of common tracking sites
and advertisement sites.

But, I haven't tested the speed (simply because it hasn't been an
issue, and I have as large a hosts file as anyone here).

On Sat, 30 Aug 2014 15:56:19 +0200, J.O. Aho wrote:

Hosts file has the negative side, you need to copy it to each and every
machine you have, including portable devices

Portable devices require root, unfortunately.

But, how can copying a text file to Windows, Linux, and Mac be harder
than trying to get a huge number of programs to perform the same task
to work on all these platforms?

Besides, the hosts file works for all protocols, whereas most of these
other methods work only for some protocols.

There's no way there is any easier yet more complete & yet totally
portable method than the hosts file method. Yes, there *are* other
methods, but let's keep this thread to just the hosts file method.

On 08/30/2014 01:31 PM, Ned Turnbull wrote:
On Sat, 30 Aug 2014 15:56:19 +0200, J.O. Aho wrote:

the hosts file was never intended for this amount of hosts and this
makes things slower...

On Linux, it does not seem to have any deleterious effect on speed.

In fact, one could argue the machine is *faster* since it doesn't
waste time visiting any number of a hosts of common tracking sites
and advertisement sites.

But, I haven't tested the speed (simply because it hasn't been an
issue, and I have as large a hosts file as anyone here).


not for speed because it wouldn't really be any smaller but i wonder if
there'd be any point in translating them to the IP #'s?

On Sat, 30 Aug 2014 17:28:09 +0000, Ned Turnbull wrote:

I found that Firefox just needs to be closed in order to flush
its cache of hosts. Or, you have to wait a rather long time,

BTW, any web site that brought up those obnoxious "do you really
want to close" windows, was also put in that hosts file.

Once Firefox was closed and opened, you never see that message
again. If you were impatient, you could go to about:config
and turn off javascript, and then close the window.

This is one thing about Firefox caching that has been a pain,
but the two solutions are easy enough:
a) Turn off javascript temporarily (just to close the window)
b) Wait for the cache to flush (and then you can close the window)

| I certainly wouldn't want a HOSTS file that big. The vast majority
| of the entires are bound to be sites I'll never visit, either first-
| party or third-party.
|
| Here are just the ten thousand sites that I've visited that I found
| obnoxious. http://pastebin.com/WJy5WH2c
|

Another option, which you may already know about,
is to block 3rd-party images. In Firefox/Pale Moon
that used to be an option in the settings. With the
corrupting influence from Google it was changed and
hidden. (Mozilla get nearly all of their excessive
income from Google.) But the setting is still there in
about:config
set (create if necessary): permissions.default.image 3

That also seems to block other external files. It can
make some websites ugly, in cases where a CSS file and/or
images are loaded from a different domain. But for the
most part it works OK. The vast majority of ads are coming
from 3rd-party spyware servers like Google/Doubleclick, so
with 3rd-party images blocked you allow honest ads -- the
ones that are actually on the website you chose to visit --
and you block spyware ads.

There are also the issues of script and iframes. If you
only care about blocking ads then that won't matter, and
you might not want to put up with the hassle. But if you
care about being tracked, take a look at this code in the
pastebin page you linked:

iframe src="//www.facebook.com/plugins/like.php?

Code like that is in most commercial webpages. It's loading
a webpage from Facebook into a iframe, which is essentially
a second browser window. What you see is just a small Facebook
button. If you block 3rd-party images you might not even
see that. But technically you've visited Facebook. The parameters
in the page request tell information about where you're coming
from, and by requesting the page you send your IP address,
userAgent, etc to Facebook. That means that Facebook is
following you all around the Internet, even if you've never visited
their website. If you have a Facebook account then their iframes
allow them to know what you're doing when you're not logged in.
Even if you just allow first-party cookies they can probably know
who you are easily. The iframe they use makes their hidden webpage
a page that you "chose" to visit, so any cookies loaded from it
are first-party. But that's just icing on the cake for their datamining
operation. Your IP is probably enough for them to figure out who you
are, and to track your movements online.

Your massive HOSTS file is blocking at least 5 domains from
Tomshardware.com, which you may never visit. It's blocking
image servers from various countries. It even blocks f**k.org
and zerofreepopcorn.com, whatever they are. But it doesn't
block Facebook. It doesn't even block google-analytics.com,
which is tracking you from the vast majority of webpages you
visit, including your pastebin page. That's the trouble with a
giant HOSTS file. It's likely to be 98% irrelevant, and while
you're blocking the oddball ad server from some obscure page
you'll probably never visit, you're not necessarily blocking the
sites that matter most.

Also worth a try is Acrylic, which is a free DNS server program.
It acts as a proxy and has it's own HOSTS file that allow wildcards.
So you can block things like *.doubleclick.net and *.doubleclick.com
to block all Google/Doubleclick ads. The normal HOSTS file requires
adding each possible subdomain. If you look at your HOSTS file
you'll see a great deal of redundancy due to that problem. Ad
servers can just keep changing the subdomain to thwart your
HOSTS file. You might have entries for 200 Doubleclick subdomains,
but you don't have an entry for the one they might create next week.

On 8/29/2014 6:46 PM, Ned Turnbull wrote:
Where can I post a 25,000 line text file for others to benefit?

I have a fantastic hosts file, improved over years, always adding
the MVP hosts file to it, and adding about a thousand obnoxious
domains that I've run into such that I almost never see a valid
popup browser (they pop up, but they are all unfound). I never
see in-page ads either.

Yes, I know that many of you use pop up blockers, and hence you have
no need for a great hosts file. Yes I know many of you use noscript,
and ghostery and all sorts of spyware blasters.

This isn't about that.
This is simply about the hosts text file. Period.
And how to upload it so that others can benefit from my effort.

All this is asking is *where* I can post my excellent HOSTS file
so that others may benefit from using it?

It's looooooong (it's almost 25K lines long!).

I just want to post it, as a text file, so others can use it and
improve it. What location do you suggest?


You need to verify that the additions you have put
on your file over the years are still relevant. There
is a program available for download that will check
each and every one, and it takes hours, if not days,
to check as many as you have. I'm sorry but I can't
recall the name of this utility.

On Sat, 30 Aug 2014 08:10:33 -0700, HASM wrote:

Lots of them don't resolve at all

I have never tested them after I put them into that file,
so, that's a good idea.

Is there a decent script, existing, which will simply test
if they resolve, and then spit out those that do versus
those that don't?

We could clean it up simply by running *that* script.

On Sat, 30 Aug 2014 14:09:49 -0400, Mayayana wrote:

Your massive HOSTS file is blocking at least 5 domains from
Tomshardware.com,

We should note that those domains are being blocked by
the *original* MVP Hosts file, downloaded yesterday:
http://winhelp2002.mvps.org/hosts.htm

$ grep -i tomshardware HOSTS
0.0.0.0 tracking.tomshardware.com
0.0.0.0 tomshardware.fr.intellitxt.com
0.0.0.0 tomshardware.se.intellitxt.com
0.0.0.0 tomshardware.us.intellitxt.com

So, we'd have to ask *them* why they are blocking that domain.

I find that Privoxy plus NoScript blocks 99.9% of all advertising with
no need for any configuration.
--
John Hasler

Dancing Horse Hill
Elmwood, WI USA

On Sat, 30 Aug 2014 14:09:49 -0400, Mayayana wrote:

It even blocks f**k.org and zerofreepopcorn.com,
whatever they are...

Hmmm.... I may be confused ... (or you?)...

Maybe you're talking about a *different* hosts file?
I just grep'd my /etc/hosts for that, and it isn't there.

$ grep zerofreepopcorn /etc/hosts
$ grep -i popcorn /etc/hosts

It's not even in the MVP HOSTS file that I had downloaded
yesterday:

$ grep zerofreepopcorn /tmp/HOSTS
$ grep -i popcorn /tmp/HOSTS

Am I confused, or are you confused?

On Sat, 30 Aug 2014 14:09:49 -0400, Mayayana wrote:

But it doesn't
block Facebook. It doesn't even block google-analytics.com,
which is tracking you from the vast majority of webpages you
visit, including your pastebin page.

My /etc/hosts does block *some* domains with facebook in
the domain name, e.g.,
$ grep -i facebook /etc/hosts
127.0.0.1 ads.ak.facebook.com
127.0.0.1 creative.ak.facebook.com
127.0.0.1 facebookinc.122.2o7.net
127.0.0.1 facebook-repto1040s2.ahlamountada.com

Of those, the first three are in the MVP HOSTS file, so,
for whatever reason, I must have run into the fourth and
blocked it myself.

The nice thing about a HOSTS file though, is that if YOU
want to block www.facebook.com, you can.