Wget question which only an expert can answer (obtain vpn config files from vpngate.net)

Paul said:

wget.exe --wait=3 --random-wait -r -I /en,/common -nc -nd http://www.vpngate.net/ -e robots=off

I created a folder first and started the process there, so
it wouldn't make a mess.

You might want to add a parameter to log all the URLs used
in that command, for later analysis (have wget dump its log
to a text file) . I didn't catch where the
index.html file came from, and it scrolled off my
Command Prompt window.

Hello Paul,
Thanks for the help as this would help others too, in two big ways:
a. They can use the same VPN themselves, and,
b. The syntax is sticky so what we learn is generally useful

I ran your command which is downloading files as we speak.
The index.html was the first file it tried to download.

C:\tmp\test2wget.exe --wait=3 --random-wait -r -I /en,/common -nc -nd
http://www.vpngate.net/ -e robots=off
-------------- here is the response
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\apps\os\gnu_get/etc/wgetrc
--2016-12-25 17:05:55-- http://www.vpngate.net/
Resolving www.vpngate.net... 130.158.75.40, 130.158.75.35, 130.158.75.42,
....
Connecting to www.vpngate.net|130.158.75.40|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: /en/ [following]
--2016-12-25 17:06:00-- http://www.vpngate.net/en/
Reusing existing connection to www.vpngate.net:80.
HTTP request sent, awaiting response... 200 OK
Length: 194506 (190K) [text/html]
Saving to: `index.html'

100%[======================================] 194,506 21.5K/s in 8.6s

2016-12-25 17:06:08 (21.5 KB/s) - `index.html' saved [194506/194506]

--2016-12-25 17:06:13-- http://www.vpngate.net/en/about.aspx
Reusing existing connection to www.vpngate.net:80.
HTTP request sent, awaiting response... 200 OK
Length: 8199 (8.0K) [text/html]
Saving to: `about.aspx'

100%[======================================] 8,199 --.-K/s in
0.007s

2016-12-25 17:06:14 (1.24 MB/s) - `about.aspx' saved [8199/8199]

------------- and its still working

I will let you know what it downloads.
So far it is a ton of files starting with "about" and "howto" but I'll let
it chunk away.

Paul said:

And what you're doing, is obtaining about 170 randomly
selected database entries out of 7700 of them.

Hello Paul.
I'm impressed with your acumen!

I have been obtaining these VPN files for about a year now, so I am
impressed that you know what you wrote here since it agrees mostly with my
long-term observations while using this free VPN service.

I don't know where you got the 7,700 (as I thought it was more like 5
thousand, but your number is close enough to mine as to be the correct
figure).

They only show a couple hundred (or whatever) at any one time of all the
files available, which is why keeping the old files is useful (because they
often work).

So rather
than being "old" and "new", they are only showing you
a subset of all entries available.

While I completely agree that they're only showing 170 out of 7,700 (which
is only about 2% of the public VPN servers they have in use), it's slightly
more complicated than that since what they do show you today will have
something like half of them being stale by tomorrow, where another half of
those will be stale the next day, and so on.

However, some of those that go stale, if you keep them around, will turn
back on again!

I think it's easy for them to do this since they require each of the
servers to use their software. They probably just change the ports but I
don't know that for a fact.

They must have some
rules as to what portion of the database they "show" per day,
to prevent abuses (Denial Of Service against each VPN host).

You are astute in that their whole approach is *different* than all the
other public VPN services, in that they are trying to foil censors.

That works perfectly, by the way, for my use, since the spammers stay away
from this site (it seems), so, the legitimate sites (like Google,
Craigslist, forum web sites, Amazon, etc.) don't have them in a blocklist
like they do all the Tor exit notes and all the known vpn services.

It would work for a spammer too, but my guess is that the fact the service
changes so frequently would drive spambots nuts, so, that's why it works
(it's designed to foil censors, but it foils spammers by the same methods).

Of course, that is why their use model is to download a file today, and see
if it works. If it doesn't work then download another one today, until you
get one that works.

That's their use model, and I'm just trying to make that use model more
efficient.

There is also a notation that says some information is
randomly corrupted, for protection purposes.

Yes. What that means, I think, is simply that some files are using
arbitrary IP addresses of some innocent machine, which won't work on VPN so
it doesn't hurt that machine.

The point, I think, is that censors can't block every IP address that is in
these files because some IP addresses (we can presume) have nothing to do
with this VPN service.

At least that is how I interpret the fact that they sprinkle bad files
among the good ones (which I can easily see in practice because one out of
about every 10 files downloaded just one minute ago will be stale from the
start).

I presume if you work at this long enough (a couple years),
you might eventually obtain all the available entries.

I probably have a thousand files by now.
a. None last forever (it seems)
b. But most come and go over time

That is, they work today; they work tomorrow; but they go stale on the
third day; but they come back alive on the tenth day (or whatever).

But
without knowledge of how they keep track, it's hard to say.
They might try to use a client cookie, but a determined
attacker would just use a botnet to milk the thing.

I would think that a state-sponsored attacker would have the resources to
block the service effectively - but it's not really my gig to figure out
how to block them (since I'm not a state sponsored censor).

I'm just a user who is manually downloading the files.

I should mention that if you use their "softether" vpn client (instead of
the Sourceforge vpn client), it automatically gives you valid files to
choose from.
http://www.softether.org/

I used that softether software for about six months, and I found that the
connections were tremendously more flaky using the softether software than
when using openvpn open-source software.
http://www.softether.org/1-features

So, while SoftEther certainly makes things super *easy*, it basically
stinks from a reliability standpoint. So I went back to the OpenSource
OpenVPN client.

Hazuki Nakamura wrote:
Paul said:

And what you're doing, is obtaining about 170 randomly
selected database entries out of 7700 of them.

Hello Paul.
I'm impressed with your acumen!

I have been obtaining these VPN files for about a year now, so I am
impressed that you know what you wrote here since it agrees mostly with my
long-term observations while using this free VPN service.

I don't know where you got the 7,700 (as I thought it was more like 5
thousand, but your number is close enough to mine as to be the correct
figure).

They only show a couple hundred (or whatever) at any one time of all the
files available, which is why keeping the old files is useful (because they
often work).

So rather
than being "old" and "new", they are only showing you
a subset of all entries available.

While I completely agree that they're only showing 170 out of 7,700 (which
is only about 2% of the public VPN servers they have in use), it's slightly
more complicated than that since what they do show you today will have
something like half of them being stale by tomorrow, where another half of
those will be stale the next day, and so on.

However, some of those that go stale, if you keep them around, will turn
back on again!

I think it's easy for them to do this since they require each of the
servers to use their software. They probably just change the ports but I
don't know that for a fact.

They must have some
rules as to what portion of the database they "show" per day,
to prevent abuses (Denial Of Service against each VPN host).

You are astute in that their whole approach is *different* than all the
other public VPN services, in that they are trying to foil censors.

That works perfectly, by the way, for my use, since the spammers stay away
from this site (it seems), so, the legitimate sites (like Google,
Craigslist, forum web sites, Amazon, etc.) don't have them in a blocklist
like they do all the Tor exit notes and all the known vpn services.

It would work for a spammer too, but my guess is that the fact the service
changes so frequently would drive spambots nuts, so, that's why it works
(it's designed to foil censors, but it foils spammers by the same methods).

Of course, that is why their use model is to download a file today, and see
if it works. If it doesn't work then download another one today, until you
get one that works.

That's their use model, and I'm just trying to make that use model more
efficient.

There is also a notation that says some information is
randomly corrupted, for protection purposes.

Yes. What that means, I think, is simply that some files are using
arbitrary IP addresses of some innocent machine, which won't work on VPN so
it doesn't hurt that machine.

The point, I think, is that censors can't block every IP address that is in
these files because some IP addresses (we can presume) have nothing to do
with this VPN service.

At least that is how I interpret the fact that they sprinkle bad files
among the good ones (which I can easily see in practice because one out of
about every 10 files downloaded just one minute ago will be stale from the
start).

I presume if you work at this long enough (a couple years),
you might eventually obtain all the available entries.

I probably have a thousand files by now.
a. None last forever (it seems)
b. But most come and go over time

That is, they work today; they work tomorrow; but they go stale on the
third day; but they come back alive on the tenth day (or whatever).

But
without knowledge of how they keep track, it's hard to say.
They might try to use a client cookie, but a determined
attacker would just use a botnet to milk the thing.

I would think that a state-sponsored attacker would have the resources to
block the service effectively - but it's not really my gig to figure out
how to block them (since I'm not a state sponsored censor).

I'm just a user who is manually downloading the files.

I should mention that if you use their "softether" vpn client (instead of
the Sourceforge vpn client), it automatically gives you valid files to
choose from.
http://www.softether.org/

I used that softether software for about six months, and I found that the
connections were tremendously more flaky using the softether software than
when using openvpn open-source software.
http://www.softether.org/1-features

So, while SoftEther certainly makes things super *easy*, it basically
stinks from a reliability standpoint. So I went back to the OpenSource
OpenVPN client.


I got a transfer to work here.

https://s24.postimg.org/nmynhw0jp/wget_single_file.gif

I suspect the SID field in the URL, is a time based identifier.
After a certain period of time, the file can no longer be fetched.

So from the time you visit and generate an index file, there
is a time-based behavior on how long the individual ovpn files
listed in the index will be present.

I tried to fetch files using the first index.html I fetched,
and those no longer work. By fetching another index.html,
those references are valid. And that's what I used
for the example in quotes in the picture.

Paul

Paul said:

I got a transfer to work here.

https://s24.postimg.org/nmynhw0jp/wget_single_file.gif

I suspect the SID field in the URL, is a time based identifier.
After a certain period of time, the file can no longer be fetched.

So from the time you visit and generate an index file, there
is a time-based behavior on how long the individual ovpn files
listed in the index will be present.

I tried to fetch files using the first index.html I fetched,
and those no longer work. By fetching another index.html,
those references are valid. And that's what I used
for the example in quotes in the picture.

I was also able to get "something" to download when I munged the index.html
file but what I got, I think, was what you got, I think, which is the
precursor file and not the openvpn *.ovpn file.

The precursor file contains the full URL to the *.ovpn file though.

For example, using your previous command...
C:\tmp\test2wget.exe --wait=3 --random-wait -r -I /en,/common -nc -nd
http://www.vpngate.net/ -e robots=off

Here is what the directory contained:
http://i.cubeupload.com/sRc4y0.jpg

Adding the extension *.txt* to the first file:


I can open it upo to find that those files are HTML files which contain the
link to the *.ovpn files that we want in the end.

a
href="/common/openvpn_download.aspx?sid=1482719400319&tcp=1& amp;host=jrcollado1987.opengw.net&port=443&amp ;hid=5923234&/vpngate_jrcollado1987.opengw.net_tcp_443.ovpn"

Specifically the ovpn file we want is:
vpngate_jrcollado1987.opengw.net_tcp_443.ovpn

Note that there are usually 2 and often 4 different ovpn files for any
given VPN server, which, for our purposes are essentially the same, but for
explanation, they a
1. The ovpn file using a hostname so DNS is needed, for TCP
2. The ovpn file using a hostname so DNS is needed, for UDP
3. The ovpn file using an IP address (so no DNS is needed), for TCP
4. The ovpn file using an IP address (so no DNS is needed), for UDP

For our purposes, any one of those 4 files is sufficient (i.e., we don't
need all four; we just need one).

Hazuki Nakamura wrote:
Paul said:

I got a transfer to work here.

https://s24.postimg.org/nmynhw0jp/wget_single_file.gif

I suspect the SID field in the URL, is a time based identifier.
After a certain period of time, the file can no longer be fetched.

So from the time you visit and generate an index file, there
is a time-based behavior on how long the individual ovpn files
listed in the index will be present.

I tried to fetch files using the first index.html I fetched,
and those no longer work. By fetching another index.html,
those references are valid. And that's what I used
for the example in quotes in the picture.

I was also able to get "something" to download when I munged the index.html
file but what I got, I think, was what you got, I think, which is the
precursor file and not the openvpn *.ovpn file.

The precursor file contains the full URL to the *.ovpn file though.

For example, using your previous command...
C:\tmp\test2wget.exe --wait=3 --random-wait -r -I /en,/common -nc -nd
http://www.vpngate.net/ -e robots=off

Here is what the directory contained:
http://i.cubeupload.com/sRc4y0.jpg

Adding the extension *.txt* to the first file:


I can open it upo to find that those files are HTML files which contain the
link to the *.ovpn files that we want in the end.

a
href="/common/openvpn_download.aspx?sid=1482719400319&tcp=1& amp;host=jrcollado1987.opengw.net&port=443&amp ;hid=5923234&/vpngate_jrcollado1987.opengw.net_tcp_443.ovpn"

Specifically the ovpn file we want is:
vpngate_jrcollado1987.opengw.net_tcp_443.ovpn

Note that there are usually 2 and often 4 different ovpn files for any
given VPN server, which, for our purposes are essentially the same, but for
explanation, they a
1. The ovpn file using a hostname so DNS is needed, for TCP
2. The ovpn file using a hostname so DNS is needed, for UDP
3. The ovpn file using an IP address (so no DNS is needed), for TCP
4. The ovpn file using an IP address (so no DNS is needed), for UDP

For our purposes, any one of those 4 files is sufficient (i.e., we don't
need all four; we just need one).

I think I see where the wget crawl is getting the second URL.

A file like this (from http://www.vpngate.net/en) :

tcp=1826&udp=1676&sid=1482703465775&hid=5075681

has a line in it like this (pointing to http://www.vpngate.net/common):

a href="/common/openvpn_download.aspx?sid=1482703465775&udp=1& amp;
host=vpn944483068.opengw.net&port=1676&hid =5075681&
/vpngate_vpn944483068.opengw.net_udp_1676.ovpn"

Now, the "&" must be converted to just "&" before usage in
a wget call. Then the download will happen and the resulting file is

openvpn_download.aspx@sid=1482703465775&udp=1&
host=vpn944483068.opengw.net&port=1676&hid=5075681 &
%2Fvpngate_vpn944483068.opengw.net_udp_1676.ovpn

So it's "index.html" to "html file" to "download ovpn file".

*******

This is another picture of a command prompt example. I opened
the file from the first call, found the string, replaced the
ampersand stuff, then pasted it into Command Prompt for the
second command. The /en contains the HTML file, the /common
contains the ovpn.

https://s23.postimg.org/pthzttsrv/wget_twice.gif

Now you're getting closer to writing a script :-)

Paul

Paul said:

A file like this (from http://www.vpngate.net/en) :

tcp=1826&udp=1676&sid=1482703465775&hid=5075681

has a line in it like this (pointing to http://www.vpngate.net/common):

a href="/common/openvpn_download.aspx?sid=1482703465775&udp=1& amp;
host=vpn944483068.opengw.net&port=1676&hid =5075681&
/vpngate_vpn944483068.opengw.net_udp_1676.ovpn"

Now, the "&" must be converted to just "&" before usage in
a wget call. Then the download will happen and the resulting file is

openvpn_download.aspx@sid=1482703465775&udp=1&
host=vpn944483068.opengw.net&port=1676&hid=5075681 &
%2Fvpngate_vpn944483068.opengw.net_udp_1676.ovpn

So it's "index.html" to "html file" to "download ovpn file".

Yup. That's it.

1. The index.html points to a couple hundred html files
2. Each of those html files points to from 1 to 4 ovpn files
3. So for each html file we can take the first of those ovpn file links

The first question is whether wget is capable of doing this with regular
expressions (which I'm sure it is).

The second question is whether the Windows wget is capable of the regular
expressions that are needed.

If wget is incapable of getting these files that would be shocking, because
that would be a Windows limitation.

Paul said:

This is another picture of a command prompt example. I opened
the file from the first call, found the string, replaced the
ampersand stuff, then pasted it into Command Prompt for the
second command. The /en contains the HTML file, the /common
contains the ovpn.

https://s23.postimg.org/pthzttsrv/wget_twice.gif

Now you're getting closer to writing a script :-)

Yes. That ovpn text file you have downloaded is the final desired result!
https://s23.postimg.org/pthzttsrv/wget_twice.gif

If you have the open-source openvpn desktop client installed, then you can
just doubleclick on that ovpn file, and voila! You're on VPN!

https://openvpn.net/index.php/open-s...downloads.html

Hazuki Nakamura wrote:
Paul said:

A file like this (from http://www.vpngate.net/en) :

tcp=1826&udp=1676&sid=1482703465775&hid=5075681

has a line in it like this (pointing to http://www.vpngate.net/common):

a href="/common/openvpn_download.aspx?sid=1482703465775&udp=1& amp;
host=vpn944483068.opengw.net&port=1676&hid =5075681&
/vpngate_vpn944483068.opengw.net_udp_1676.ovpn"

Now, the "&" must be converted to just "&" before usage in
a wget call. Then the download will happen and the resulting file is

openvpn_download.aspx@sid=1482703465775&udp=1&
host=vpn944483068.opengw.net&port=1676&hid=5075681 &
%2Fvpngate_vpn944483068.opengw.net_udp_1676.ovpn

So it's "index.html" to "html file" to "download ovpn file".

Yup. That's it.

1. The index.html points to a couple hundred html files
2. Each of those html files points to from 1 to 4 ovpn files
3. So for each html file we can take the first of those ovpn file links

The first question is whether wget is capable of doing this with regular
expressions (which I'm sure it is).

The second question is whether the Windows wget is capable of the regular
expressions that are needed.

If wget is incapable of getting these files that would be shocking, because
that would be a Windows limitation.

The only reason for analysing it at this level of
detail, is to see if the process could be made to
go faster or be more efficient in terms of bandwidth.

The first command I got working, is sufficient to give
you the materials needed for ovpn. This screwing around
I'm doing, is a precursor step to writing a script.

Paul

En el artículo , Hazuki Nakamura
escribió:

If wget is incapable of getting these files that would be shocking, because
that would be a Windows limitation.

It's Windows shrug

I'm sure if you installed Cygwin and used GNU wget you would achieve
what you want to do much more quickly and with less hassle.

Another possibility might be PowerShell, though that's where I bow out
with little experience.

It's interesting reading your 'stream of consciousness' posts while you
try and get your system to bend to your will

--
(\_/)
(='.'=) systemd: the Linux version of Windows 10
(")_(")

Mike Tomlinson said:

I'm sure if you installed Cygwin and used GNU wget you would achieve
what you want to do much more quickly and with less hassle.

I have used cygwin in the past, and I come from a (limited) initial
initiation on Linux, where every command was sed/awk/grep it seems, so,
that's why, for example, I use VIM to (manually) strip out all but the
desired lines out of the three files:
1. Initial HTML (at vpngate.net)
2. VPN-server HTML (one for each VPN server)
3. Payload HTML (up to 4 ovpn files per page, of which we only need one)

It's interesting reading your 'stream of consciousness' posts while you
try and get your system to bend to your will

I hope you meant that in a good way, where your apt description is apropos,
in that I'm struggling along, step by step, to get wget to download what
is, in the final tally, a static set of variables.

Therefore, I knew, from the start, that this wget Windows question, was
probably the hardest wget Windows question ever asked of this group, which
had relevance to the general user.

The relevance to the general user, if that is at all in doubt, is that once
we solve this together, all Windows users can benefit, since who can doubt
the utility of being able to utilize VPN when needed, from almost any
country in the world, especially a VPN which is unique in that:
a) It purports to foil state-sponsored censors, and,
b) From experience, I know that it is generally not banned by legit sites
like all Tor exit nodes are, so it has utility for the common user indeed!

That this is perhaps the most difficult practical Windows wget question
ever posed in the long history of this newsgroup alone wouldn't give the
question enough value to resolve; but the added utility to everyone who
reads this gives the question a social value that merits the technical
challenges still to be resolved.

Paul said:

The first command I got working, is sufficient to give
you the materials needed for ovpn. This screwing around
I'm doing, is a precursor step to writing a script.

I appreciate the help and advice.

Even though my first machine was a Linux machine, I'm now totally on
Windows. On Linux, our daily commands were easily scripted, which is to say
"strung together" (for that's my scripting ability in a nutshell).

So, I can string together commands that work in batch mode.

On Windows, I lose sed/awk/grep, etc. (I don't have cygwin installed), so
that limits my abilities tremendously.

However, I can manually edit a file in VIM using regular expresssions,
which, while not having the power of sed/awk/grep, do have some power.

So at this point, thanks almost entirely to your help and advice, I can
follow your instructions for *manually* (1) downloading the home-page HTML
file, to then manually (2) strip out the 170 or so individual vpn-server
pages, and then finally, again manually (3) grab the first *.ovpn payload
of each vpn-server page to put it into a long 170-line wget script which
calls wget individually 170 times with a sleep in between each.

I knew this was a difficult task for wget from the start, but even I (who
had spent hours trying to get wget to work - blundering along mostly) knew
this was a question for wget experts or Windows experts, and not for the
novice:

The three levels, as Paul outlined them aptly, seem to be:
1. Initial HTML (at vpngate.net)
2. VPN-server HTML (one for each VPN server)
3. Payload HTML (up to 4 ovpn files per page, of which we only need one)

If I can get wget to do the sed/awk/grep-style stripping of the payload
links, that is where I'm concentrating my efforts at this very moment.

En el artículo , Hazuki Nakamura
escribió:

I hope you meant that in a good way

absolutely, yes. It's the season of goodwill after all

Apologies if you thought otherwise.

the added utility to everyone who
reads this gives the question a social value that merits the technical
challenges still to be resolved.

Agreed completely.

--
(\_/)
(='.'=) systemd: the Linux version of Windows 10
(")_(")

Paul said:

On Windows 10, you *do* have sed/awk/grep.

I'm on Windows 7 and XP but I figured that Windows 10 users are on the
latest and greatest OS so your information is useful to all.

I think sticking with wget 1.11.4
(same version as bundled in wsusoffline) and using
Gawk 3 from GNUWIN32, you could cobble together
something.

I'm positive I can cobble together something, so that is not the problem.
The problem is/was getting wget, alone, to do the job.

I'm still convinced wget can do the job, which is why I asked in the first
place, but, as you can all see, it's not an easy answer to the simple
question.

That's if you actually like awk programming
of course.

AWK works fine when we are faced with field-delimited information which the
first and middle HTML file is filled with.

I have moments I like it, and moments
I don't like it.

I have those same moments with VIM!

In hindsight, I should have learned emacs, but the finger memory won't let
me put my hands on another girl.

Here's a script for you to play with.

BEGIN {
fetch_index = "wget.exe http://www.vpngate.net/en --output-document=-"
}

Oh my. Detailed indeed.
I will try it out but I will need cygwin at the very least, first, before
some of those commands will work on Windows 7 or XP (which is what I'm on).

But the goal always was to help everyone, so, your information would also
help the Windows 10 users.

I'm going to battle wget a bit more to see how I can get it to dig into the
second page and then pull out the ovpn files from that second page.

Hazuki Nakamura wrote:
Paul said:

On Windows 10, you *do* have sed/awk/grep.

I'm on Windows 7 and XP but I figured that Windows 10 users are on the
latest and greatest OS so your information is useful to all.

I think sticking with wget 1.11.4
(same version as bundled in wsusoffline) and using
Gawk 3 from GNUWIN32, you could cobble together
something.

I'm positive I can cobble together something, so that is not the problem.
The problem is/was getting wget, alone, to do the job.

I'm still convinced wget can do the job, which is why I asked in the first
place, but, as you can all see, it's not an easy answer to the simple
question.

That's if you actually like awk programming
of course.

AWK works fine when we are faced with field-delimited information which the
first and middle HTML file is filled with.

I have moments I like it, and moments
I don't like it.

I have those same moments with VIM!

In hindsight, I should have learned emacs, but the finger memory won't let
me put my hands on another girl.

Here's a script for you to play with.

BEGIN {
fetch_index = "wget.exe http://www.vpngate.net/en --output-document=-"
}

Oh my. Detailed indeed.
I will try it out but I will need cygwin at the very least, first, before
some of those commands will work on Windows 7 or XP (which is what I'm on).

But the goal always was to help everyone, so, your information would also
help the Windows 10 users.

I'm going to battle wget a bit more to see how I can get it to dig into the
second page and then pull out the ovpn files from that second page.

I specifically selected that method, so you *would* be able to
do it on WinXP. After all, I *wrote and tested* the script
on WinXP. Why wouldn't it work there.

And the Bash shell on Win10 can give you tools to play with.
Lots of tools. If you need them. The Bash subsystem is
provided by Canonical under contract.

Paul