Merge text files & then sort unique in DOS

Merging text files is easy in DOS
for %f in (*.txt) do type "%f" output.txt

But how do you sort unique in DOS?

On Tue, 4 Sep 2018 10:16:57 -0500, Grease Monkey wrote:
Merging text files is easy in DOS
for %f in (*.txt) do type "%f" output.txt

But how do you sort unique in DOS?

You can sort the contents of a text file using the SORT program. e.g.

type file.txt | sort

But removing duplicate lines is not possible using SORT, because it simple
doesn't have such capability built it.

So, you'll have to check for duplicates manually from a batch file. e.g.

@echo off
setlocal enabledelayedexpansion
:: sort contents
type output.txt | sort sorted.txt
set line=
del unique.txt 2nul
for /f "delims=" %%A in ('type sorted.txt') do (
if not "%%A" == "!line!" (
echo %%A unique.txt
set line=%%A
)
)

Note that this is problematic if the file contains double quote (")
character. In this case, I'd suggest using JScript.

Replying to 09/04 JJ

You can sort the contents of a text file using the SORT program. e.g.

type file.txt | sort

Yep. If it were Linux, it would simply be piped to "sort -u".

But removing duplicate lines is not possible using SORT, because it simple
doesn't have such capability built it.

Yep. That's why I asked the question.

I can already sort albeit manually using regular expressions in gvim but
I'm trying to batch merge text files unique in DOS outside of gvim.

So, you'll have to check for duplicates manually from a batch file. e.g.

@echo off
setlocal enabledelayedexpansion
:: sort contents
type output.txt | sort sorted.txt
set line=
del unique.txt 2nul
for /f "delims=" %%A in ('type sorted.txt') do (
if not "%%A" == "!line!" (
echo %%A unique.txt
set line=%%A
)
)

Note that this is problematic if the file contains double quote (")
character. In this case, I'd suggest using JScript.

Thanks for that suggestion. I'm manually using a regexp from here
http://vim.wikia.com/wiki/Uniq_-_Rem...uplicate_lines
https://www.regular-expressions.info...catelines.html

But I was hoping for a DOS command that sorted uniquely.

Grease Monkey wrote:
Replying to 09/04 JJ

You can sort the contents of a text file using the SORT program. e.g.

type file.txt | sort

Yep. If it were Linux, it would simply be piped to "sort -u".

But removing duplicate lines is not possible using SORT, because it simple
doesn't have such capability built it.

Yep. That's why I asked the question.

I can already sort albeit manually using regular expressions in gvim but
I'm trying to batch merge text files unique in DOS outside of gvim.

So, you'll have to check for duplicates manually from a batch file. e.g.

@echo off
setlocal enabledelayedexpansion
:: sort contents
type output.txt | sort sorted.txt
set line=
del unique.txt 2nul
for /f "delims=" %%A in ('type sorted.txt') do (
if not "%%A" == "!line!" (
echo %%A unique.txt
set line=%%A
)
)

Note that this is problematic if the file contains double quote (")
character. In this case, I'd suggest using JScript.

Thanks for that suggestion. I'm manually using a regexp from here
http://vim.wikia.com/wiki/Uniq_-_Rem...uplicate_lines
https://www.regular-expressions.info...catelines.html

But I was hoping for a DOS command that sorted uniquely.

http://gnuwin32.sourceforge.net/packages/coreutils.htm

You want the binaries download and the dependencies download.
Put these three files in the same directory. The latter two
come from the dep zip file.

sort.exe
libiconv2.dll
libintl3.dll

Check it's working:

sort --help

Run a test case. The input file is

delta
bravo
alpha
delta
bravo
alpha

sort -u in.txt out.txt

alpha
bravo
delta

You should also be able to pipe it.

type in.txt | sort -u

HTH,
Paul

Grease Monkey wrote:
Replying to 09/04 JJ

You can sort the contents of a text file using the SORT program. e.g.

type file.txt | sort

Yep. If it were Linux, it would simply be piped to "sort -u".

But removing duplicate lines is not possible using SORT, because it simple
doesn't have such capability built it.

Yep. That's why I asked the question.
[...]
But I was hoping for a DOS command that sorted uniquely.

Paul already gave a solution for a Linux-like (GNU) sort(1).

Since you seem to be used to Linux, I strongly recommend to install
Cygwin [1] and have all the Linux-like commands you like, in a 'DOS'
Command Prompt window or/and in a bash(1) window.

That way, you have the best of both worlds.

FWIW, I have a - three and a half decade - (professional) UNIX
background and have always used UNIX tools under MS-DOS/MS-Windows. I've
been using Cygwin for some 15-20 years now (see my User-Agent: header).

[1] http://cygwin.com

Grease Monkey,

Merging text files is easy in DOS
for %f in (*.txt) do type "%f" output.txt

Another way, likely faster:
copy *.txt output.txt
-or, for safity-
copy /a *.txt output.txt

But how do you sort unique in DOS?

You don't. There is no such (buildin) command available.

But Google is your friend: "xp batch remove duplicate lines" (without the
quotes).

First result ? Looks to be a keeper:
https://stackoverflow.com/questions/...from-text-file

Regards,
Rudy Wieser

On 9/4/2018 11:16 PM, Grease Monkey wrote:
Merging text files is easy in DOS
for %f in (*.txt) do type "%f" output.txt

But how do you sort unique in DOS?


MS DOS has a SORT.EXE if I remember correctly... it's such a long time
ago....

--
@~@ Remain silent! Drink, Blink, Stretch! Live long and prosper!!
/ v \ Simplicity is Beauty!
/( _ )\ May the Force and farces be with you!
^ ^ (x86_64 Ubuntu 9.10) Linux 2.6.39.3
ä¸�借貸! ä¸�è©�騙! ä¸�è³*錢! ä¸�æ�´äº¤! ä¸�打交! ä¸�打劫! ä¸�自殺! ä¸�求神! 請考慮綜æ�´
(CSSA):
http://www.swd.gov.hk/tc/index/site_...sub_addressesa

Mr. Man-wai Chang wrote:
On 9/4/2018 11:16 PM, Grease Monkey wrote:
Merging text files is easy in DOS
for %f in (*.txt) do type "%f" output.txt

But how do you sort unique in DOS?


MS DOS has a SORT.EXE if I remember correctly... it's such a long time
ago....

Why all the additional crossposted groups ?

*******

There is no UNIQUE option in MSDOS sort.

http://www.vfrazee.com/ms-dos/6.22/help/sort.htm

Paul

On 9/6/2018 1:33 AM, Paul wrote:

Why all the additional crossposted groups ?


To reach out for more people and hence more help and answers.

Anyway, I always avoid over-doing that.
--
@~@ Remain silent! Drink, Blink, Stretch! Live long and prosper!!
/ v \ Simplicity is Beauty!
/( _ )\ May the Force and farces be with you!
^ ^ (x86_64 Ubuntu 9.10) Linux 2.6.39.3
ä¸�借貸! ä¸�è©�騙! ä¸�è³*錢! ä¸�æ�´äº¤! ä¸�打交! ä¸�打劫! ä¸�自殺! ä¸�求神! 請考慮綜æ�´
(CSSA):
http://www.swd.gov.hk/tc/index/site_...sub_addressesa

Mr. Man-wai Chang wrote:
On 9/6/2018 1:33 AM, Paul wrote:

Why all the additional crossposted groups ?


To reach out for more people and hence more help and answers.

Anyway, I always avoid over-doing that.

alt.comp.hardware.pc-homebuilt === a hardware group,
over-doing it!

Paul

On 9/6/2018 8:35 PM, Paul wrote:

alt.comp.hardware.pc-homebuilt === a hardware group,
Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â* Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â* over-doing it!

Sorry, SIR!

--
@~@ Remain silent! Drink, Blink, Stretch! Live long and prosper!!
/ v \ Simplicity is Beauty!
/( _ )\ May the Force and farces be with you!
^ ^ (x86_64 Ubuntu 9.10) Linux 2.6.39.3
ä¸�借貸! ä¸�è©�騙! ä¸�è³*錢! ä¸�æ�´äº¤! ä¸�打交! ä¸�打劫! ä¸�自殺! ä¸�求神! 請考慮綜æ�´
(CSSA):
http://www.swd.gov.hk/tc/index/site_...sub_addressesa

"Paul" wrote in message news There is no UNIQUE option in MSDOS sort.
http://www.vfrazee.com/ms-dos/6.22/help/sort.htm


Paul I Use IrfanView

Click on the File Menu, select Slideshow.
A dialog allows you to select a directory
from which the files will be

Click the Sort button to choose a sort order by name, date, size or
extension.
Files may be sorted in ascending or descending order. To remove a sort,
click the no Sort option button.

Click Save filenames as TXT
and a dialog like Save As appears.
Give your slideshow a name
and click Save to have it stored
on the disk drive.

In article ,
says...

On 9/4/2018 11:16 PM, Grease Monkey wrote:
Merging text files is easy in DOS
for %f in (*.txt) do type "%f" output.txt

But how do you sort unique in DOS?


MS DOS has a SORT.EXE if I remember correctly... it's such a long time
ago....

Yes, but it doesn't have a /unique option.

Powershell can do this, as can the Bash shell if you install that.

--

Phil, London

In article ,
says...

On 9/4/2018 11:16 PM, Grease Monkey wrote:
Merging text files is easy in DOS
for %f in (*.txt) do type "%f" output.txt

But how do you sort unique in DOS?


MS DOS has a SORT.EXE if I remember correctly... it's such a long time
ago....

Yes, but it doesn't have a /unique option.

Powershell can do this, as can the Bash shell if you install that.

(only now seeing that this wasn't sent when I expected it to be!)

--

Phil, London

In article ,
says...

In article ,
says...

On 9/4/2018 11:16 PM, Grease Monkey wrote:
Merging text files is easy in DOS
for %f in (*.txt) do type "%f" output.txt

But how do you sort unique in DOS?


MS DOS has a SORT.EXE if I remember correctly... it's such a long time
ago....

Yes, but it doesn't have a /unique option.

Powershell can do this, as can the Bash shell if you install that.

(only now seeing that this wasn't sent when I expected it to be!)

Hi

I fetched a long time ago a set of unix like tools for windows.
One exe is sort.exe. I renamed it to sortX.exe to avoid a conflict with
the windows sort. It's working with XP, ... windows-10.



Microsoft Windows [Version 10.0.17134.228]
(c) 2018 Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\Reinhardcd C:\TOOLS

C:\TOOLSsortX --help
Usage: sortX [OPTION]... [FILE]...
Write sorted concatenation of all FILE(s) to standard output.

Ordering options:

Mandatory arguments to long options are mandatory for short options too.
-b, --ignore-leading-blanks ignore leading blanks
-d, --dictionary-order consider only blanks and alphanumeric
characters
-f, --ignore-case fold lower case to upper case characters
-g, --general-numeric-sort compare according to general numerical
value
-i, --ignore-nonprinting consider only printable characters
-M, --month-sort compare (unknown) `JAN' ... `DEC'
-n, --numeric-sort compare according to string numerical
value
-r, --reverse reverse the result of comparisons

Other options:

-c, --check check whether input is sorted; do not sort
-k, --key=POS1[,POS2] start a key at POS1, end it at POS 2 (origin
1)
-m, --merge merge already sorted files; do not sort
-o, --output=FILE write result to FILE instead of standard
output
-s, --stable stabilize sort by disabling last-resort
comparison
-S, --buffer-size=SIZE use SIZE for main memory buffer
-t, --field-separator=SEP use SEP instead of non- to whitespace
transition
-T, --temporary-directory=DIR use DIR for temporaries, not $TMPDIR or
c:/temp
multiple options specify multiple
directories
-u, --unique with -c: check for strict ordering
otherwise: output only the first of an
equal run
-z, --zero-terminated end lines with 0 byte, not newline
--help display this help and exit
--version output version information and exit

POS is F[.C][OPTS], where F is the field number and C the character
position
in the field. OPTS is one or more single-letter ordering options, which
override global ordering options for that key. If no key is given, use
the
entire line as the key.

SIZE may be followed by the following multiplicative suffixes:
% 1% of memory, b 1, K 1024 (default), and so on for M, G, T, P, E, Z,
Y.

With no FILE, or when FILE is -, read standard input.

*** WARNING ***
The locale specified by the environment affects sort order.
Set LC_ALL=C to get the traditional sort order that uses
native byte values.

Report bugs to .

C:\TOOLS


Maybe you find "textutils" somewhere in the web.
If not I will put it on workdownload.com

Send a mail to


Regards
Reinhard