Convert those dastardly curly quotes to straight quotes on Windows?

"harry newton" wrote

| Since almost every program edits text, I don't think we want to go down
the
| path of defining whether Notepad is a bona-fide "text editor" (because in
| my book, it's not a real text editor but in other people's books - it is -
| where I just said EVERYTHING edits text so that argument would never end).
|
| My text editor on Windows/Linux/Android is the standard vi clone.
|
| What I *meant* by *text editor* is "pure text" or "real text" or whatever
| it is called when all that formatting junk is removed leaving only the
| characters on the screen that are on the keyboard itself.

Going down the path of defining text editors would
actually be helpful. The main problem you're having is
that you don't understand the distinctions.

Text editor usually refers to plain text. No font options,
color, etc. But plain text can be of different encodings.
A text editor is used to write plain text that doesn't
need formatting. It then saves that text to file as a simple
stream of bytes. Typically each byte represents a character.
Sometimes 2 bytes represent a character.

Vi/vim/gvim is not a good text editor because it's designed
to be a coding editor. Besides the horrendous UI, it doesn't
seem to handle different encodings. That's why you're having
trouble. Anything you paste is rendered as ASCII, because Vi
is assuming that you're writing code.

Wordpad is known as a "rich text" editor, which seems to
be a Microsoft invention. It's pretty much what Word started
out as. Like HTML, Wordpad RTF files are composed of
plain ASCII text with special formatting characters and
patterns to tell the RTF rendering window how to display
font, color, etc. So like HTML, rich text displays as plain
text but is not. Word DOCs are a kind of ANSI monstrosity
with unicode sprinkled in, along with a lot of gibberish to
break compatibility with copycats. DOCX is actually a
ZIP file, containing numerous files and making heavy use of
Microsoft's current darling, XML, to specify formatting. It
looks like plain text when Word renders it, but it's a whole
different thing. (Look for the next MS Office format to
use JSON. There's always a next big thing bandwagon in
the tech world that everyone feels compelled to jump onto.)

When you print text in GIMP or another graphic editor
that's usually raster graphics. In other words, it's not text
at all. It's a picture. In Notepad the byte 65 renders as A.
In GIMP an A is actually a bitmap, or part of a bitmap. Each
pixel in the A is represented by 3 bytes representing RGB
color values. So an A in GIMP might require 30,000 bytes
to render, while an A in Notepad requires one.

I find it helps to understand some of how it all works
behind the scenes in order to make sense of issues
like the one you're having with "dastardly text".

"harry newton" wrote

| Why would NotePad++ work when MS Office failed?

You don't need either of those. Paste into Notepad
and save as UTF-8. That option appears in a dropdown
underneath the Save As textbox, in the Save As window.

He who is NotMe said on Sun, 8 Oct 2017 16:32:28 -0500:

Doe not,from memory, Paste Special-Plain Text or some equivalent do that?
Should be Unformatted Text.

Here's what I just tried to follow that suggestion as a tested solution.

1. I added a quick way to bring up a blank intermediary MS Word document.
http://wetakepic.com/images/2017/10/09/fixquote1.jpg
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\fixquote.exe
Default = C:\tmp\junk\fixquote.docx
That brings up a blank Word doc when the "fixquote" keyword is typed.
Windows +- RfixquoteEnter

2. I copied the problematic text into my clipboard from any web site:
https://practicaltypography.com/straight-and-curly-quotes.html

Here is what I copied into my Windows clipboard (also pasted below):
http://wetakepic.com/images/2017/10/09/smartquotes2f7994.jpg

Straight quotes are the two generic ver+AK0-ti+AK0-cal quo+AK0-ta+AK0-tion marks
lo+AK0-cated near the re+AK0-turn key: the straight sin+AK0-gle quote (')
and the straight dou+AK0-ble quote (").

Curly quotes are the quo+AK0-ta+AK0-tion marks used in good ty+AK0-pog+AK0-ra+AK0-phy.
There are four curly quote char+AK0-ac+AK0-ters: the open+AK0-ing sin+AK0-gle quote
(+IBg-), the clos+AK0-ing sin+AK0-gle quote (+IBk-), the open+AK0-ing dou+AK0-ble quote (+IBw-),
and the clos+AK0-ing dou+AK0-ble quote (+IB0-).

3. I pasted my clipboard "without formatting" into that Word doc:
Control +- Alt +- V[x]Unformatted text or Unformatted Unicode text
Both failed:
http://wetakepic.com/images/2017/10/09/fixquote3.jpg

In summary, pasting "unformatted" text is great to remove formatting.
But it doesn't convert those dastardly twisted quotes into straight quotes.

On Mon, 9 Oct 2017 04:18:15 +0000 (UTC), harry newton
wrote:

The reason I'm using GVIM on Windows is that vi allows for quick edits.

I tried to use Google to find "gvim quick edits" and came up empty. What
does GVIM do that other text editors don't do? What are quick edits? I
assume those two words don't have their obvious meaning, because every
text editor does that, so what is it?

"Peter Moylan" wrote

| UTF-8 is a way to express unicode using single bytes.
| Unicode-16, what's usually just referred to as unicode,
| encodes thousands of characters in 2 bytes, so each character
| can have its own specific encoding number in order to fit
| English, Russian and everything else. ASCII and ANSI use
| a one-byte-per-character encoding, except with a few
| Asian languages.
|
| Minor correction: Unicode has more than 2^16 code points, so you can't
| express it using two bytes per character. (For full coverage you'd need
| a 21-bit code.) An earlier version of Unicode did allow a 16-bit
| representation, but that's now obsolete because it didn't cover the
| characters of some languages.
|
I stand by my statement. It's true that there are various
unicode versions, but the only ones that anyone is likely to
encounter are UTF-8 in webpages and unicode-16 in computer
files. Windows runs on unicode-16, which is not UTF-16 but
rather a 2-byte-per-character system. It's hardly obsolete.
So while the bytes representing ABC would be 65-66-67 in
ASCII/ANSI/UTF-8, and probably UTF-16, they're
00-65-00-66-00-67 in unicode.
If you look at any Windows DLL in a hex editor you
can see samples of both ASCII/ANSI and unicode. Things
like the import and export tables are in ASCII, while version
info strings are stored as unicode. Unicode is also the
standard format for strings in Windows programming.

On Mon, 9 Oct 2017 04:06:35 +0000 (UTC), harry newton
wrote:

He who is J. P. Gilliver (John) said on Sun, 8 Oct 2017 16:58:57 +0100:

Word does that (by default - you can turn it off); if I type "Fred", it
will convert the quotes into the 66 and 99 form (I think it calls them
"smart quotes"). [I think it does the same with single quotes, 'Fred'.]
You can stop it doing it either by turning off the setting, or on a
one-off basis by doing an Undo (Ctrl-Z) immediately after typing the ".

I wouldn't be surprised if some web-page editing software behaves
similarly.

I tried Word ... and failed.

I was hoping I could paste into MS Word 2007 to have Word automatically
remove the curly quotes, replacing them with keyboard quotes (aka ASCII
quotes).

This article shows how to turn off the Microsoft Word default to convert
keyboard quotes into "smart quotes" (aka curly quotes).
https://support.office.com/en-us/article/Change-curly-quotes-to-straight-quotes-and-vice-versa-017963A0-BC5F-486B-9C9D-0EC511A8FB8F

Here's a screenshot of the keyboard-to-smart quotes GUI in my Word 2007:
http://wetakepic.com/images/2017/10/09/smartquotes.jpg

But a test cut-and-paste of the original article into MS Word failed to
convert the curly quotes to keyboard quotes (or ASCII quotes).
http://theverge.com/2017/10/6/16437790/iphone-8-swollen-battery-issue-apple-investigating

It didn't work:
http://wetakepic.com/images/2017/10/09/smartquotes2.jpg

Since I don't know MS Word all that well I wonder aloud whether MS Word can
convert curly quotes to keyboard quotes.

If Microsoft Word can convert curly quotes to keyboard quotes, the issue
would be solved instantly.

Does anyone know if Word can convert those curly quotes to keyboard quotes?
http://wetakepic.com/images/2017/10/09/smartquotes2.jpg

In my limited experience, the answer is yes (configurable to be
automatic) if you're typing, and no if you're pasting. Are you seeing a
different result?

Either way, once you've pasted, it's trivial to record a macro that
converts from what you don't want to what you do want. The only issue I
take with that approach is that Word's macro recorder seldom produces
VBA code as efficient as what I can create manually, but it's usually a
great start.

On 09/10/17 02:39, David E. Ross wrote:

Seehttp://www.fourmilab.ch/webtools/demoroniser/. This is a tool
that supposedly converts Microsoft "smart" characters to HTML-compatible
characters. Yes, it is 14 years old; and no, I have not tried it myself.

There's nothing wrong with using old software; it doesn't go rusty, or
anything like that. You don't even have to dust it.

The other day I had to add a feature to my FTP server where a list of
users could be sorted by username, or user type, or timestamp of last
login. To do the sort I used a Quicksort module that I had written 21
years ago. I wasn't surprised that it worked without changes. If it
worked back then, it should work now.

There is a belief in some quarters that if there have been no bug fixes
for a program in the last year or so, then it is "abandonware". That is
annoying to those of us who choose not to insert the bugs in the first
place.

--
Peter Moylan http://www.pmoylan.org
Newcastle, NSW, Australia

"Peter Moylan" wrote

| Why as much as 2 MB for a short memo? Because apparently he didn't know
| how to create a new MS-Word document, so he would take an existing
| document, delete its content, and then type the new text. What he didn't
| realise was that, with an unfortunate choice of options (it might even
| be the default choice), the document contains a record of all changes.
| This became obvious to those of us who didn't have MS-Word installed and
| had to read the raw data.
|
| What also became obvious was that he was leaking some very confidential
| data.

That seems to be a lesson that hasn't been learned
many times. I consider it a bug in MS Word that things
like user names and changes are stored. But I guess it
makes sense in the context of most Word users being
corporate employees who are using Word for work and
not for their own purposes.
If I remember correctly, that's also how the creator
of the Melissa virus
got caught. He was just an office worker with enough
knowledge of VBScript to make a mischievous macro.
What he didn't realize was that his name was embedded
in the file.

He who is Mayayana said on Mon, 9 Oct 2017 01:11:43 -0400:

| Why would NotePad+-+- work when MS Office failed?

You don't need either of those. Paste into Notepad
and save as UTF-8. That option appears in a dropdown
underneath the Save As textbox, in the Save As window.

1. I copied problematic text from an arbitrary curlyquote web site:
https://practicaltypography.com/straight-and-curly-quotes.html

Curly quotes are the quo+AK0-ta+AK0-tion marks used in good ty+AK0-pog+AK0-ra+AK0-phy.
There are four curly quote char+AK0-ac+AK0-ters: the open+AK0-ing sin+AK0-gle quote
(+IBg-), the clos+AK0-ing sin+AK0-gle quote (+IBk-), the open+AK0-ing dou+AK0-ble quote (+IBw-),
and the clos+AK0-ing dou+AK0-ble quote (+IB0-).

2. I pasted into a blank MS Word doc by typing Windows+-R fixquote Enter
and then Control+-Alt+-V "Paste Special Unformatted Unicode Text".

3. MS Word 2007 didn't have a Save as "UTF-8" but it did have "Plain Text".
And yet, the text *still* had the non-printable non-keyboard characters!
Even when I selected US-ASCII as the encoding format.
http://wetakepic.com/images/2017/10/09/fix3.jpg

4. Having not used NotePad in centuries, I pasted into Notepad this time
Start Run notepad Enter
And saved as UTF-8.
Yikes! The result was even worse!
http://wetakepic.com/images/2017/10/09/fix4.jpg

I think I don't understand the problem because the result of that was a
"plain text" file which contained utterly unprintable non-keyboard
characters.

What did I do wrongly?

On 10/8/2017 10:53 PM, harry newton wrote:
He who is Mayayana said on Mon, 9 Oct 2017 01:11:43 -0400:

| Why would NotePad+-+- work when MS Office failed?

* You don't need either of those. Paste into Notepad
and save as UTF-8. That option appears in a dropdown
underneath the Save As textbox, in the Save As window.

1. I copied problematic text from an arbitrary curlyquote web site:
* https://practicaltypography.com/straight-and-curly-quotes.html

* Curly quotes are the quo+AK0-ta+AK0-tion marks used in good
ty+AK0-pog+AK0-ra+AK0-phy. * There are four curly quote
char+AK0-ac+AK0-ters: the open+AK0-ing sin+AK0-gle quote * (+IBg-), the
clos+AK0-ing sin+AK0-gle quote (+IBk-), the open+AK0-ing dou+AK0-ble
quote (+IBw-), * and the clos+AK0-ing dou+AK0-ble quote (+IB0-).

2. I pasted into a blank MS Word doc by typing Windows+-R fixquote
Enter
* and then Control+-Alt+-V "Paste Special Unformatted Unicode Text".

3. MS Word 2007 didn't have a Save as "UTF-8" but it did have "Plain Text".
* And yet, the text *still* had the non-printable non-keyboard characters!
* Even when I selected US-ASCII as the encoding format.
* http://wetakepic.com/images/2017/10/09/fix3.jpg

4. Having not used NotePad in centuries, I pasted into Notepad this time
* Start Run notepad Enter
* And saved as UTF-8.
* Yikes! The result was even worse!
* http://wetakepic.com/images/2017/10/09/fix4.jpg

I think I don't understand the problem because the result of that was a
"plain text" file which contained utterly unprintable non-keyboard
characters.
What did I do wrongly?

Can you make a Word doc available for d/l somewhere that has all of the
characters you want to replace with one 'bad' character, space,
replacement character pair per line?

He who is Char Jackson said on Mon, 09 Oct 2017 00:22:09 -0500:

The reason I'm using GVIM on Windows is that vi allows for quick edits.

I tried to use Google to find "gvim quick edits" and came up empty. What
does GVIM do that other text editors don't do? What are quick edits? I
assume those two words don't have their obvious meaning, because every
text editor does that, so what is it?

I apologize. My words were not clear. Hence I caused unnecessary confusion.

I appreciate that you followed the implied meaning of my words, which makes
it my fault for not explaining that "vi" is all I ever wanted. GVIM just
came with the package when I searched for the best "vi" editor on Windows.

I never use the GUI in GVIM. I never use the mouse. Both hands are on the
keyboard, because all I do with GVIM is use the "vi" part of VIM.

So what I meant by "quick edits" was my own concoction of words to describe
that fact that I can cut and paste and copy and rename and delete and
reorder, etc., all using keyboard presses.

For example, I can delete a line by typing "dd" and I can yank a line by
typing "yy" and I can paste those deleted or yanked lines ten lines lower
by typing "p" after jumping down ten lines with "10j" such that the
self-concocted phrase "quick edits" means something like this sequence:

yy10jp == this is a "quick edit" yanking a line to place it 10 lines lower

On Mon, 9 Oct 2017 05:53:24 +0000 (UTC), harry newton
wrote:

He who is Mayayana said on Mon, 9 Oct 2017 01:11:43 -0400:

| Why would NotePad+-+- work when MS Office failed?

You don't need either of those. Paste into Notepad
and save as UTF-8. That option appears in a dropdown
underneath the Save As textbox, in the Save As window.

1. I copied problematic text from an arbitrary curlyquote web site:
https://practicaltypography.com/straight-and-curly-quotes.html

Curly quotes are the quo+AK0-ta+AK0-tion marks used in good ty+AK0-pog+AK0-ra+AK0-phy.
There are four curly quote char+AK0-ac+AK0-ters: the open+AK0-ing sin+AK0-gle quote
(+IBg-), the clos+AK0-ing sin+AK0-gle quote (+IBk-), the open+AK0-ing dou+AK0-ble quote (+IBw-),
and the clos+AK0-ing dou+AK0-ble quote (+IB0-).

2. I pasted into a blank MS Word doc by typing Windows+-R fixquote Enter
and then Control+-Alt+-V "Paste Special Unformatted Unicode Text".

3. MS Word 2007 didn't have a Save as "UTF-8" but it did have "Plain Text".
And yet, the text *still* had the non-printable non-keyboard characters!
Even when I selected US-ASCII as the encoding format.
http://wetakepic.com/images/2017/10/09/fix3.jpg

4. Having not used NotePad in centuries, I pasted into Notepad this time
Start Run notepad Enter
And saved as UTF-8.
Yikes! The result was even worse!
http://wetakepic.com/images/2017/10/09/fix4.jpg

I think I don't understand the problem because the result of that was a
"plain text" file which contained utterly unprintable non-keyboard
characters.

What did I do wrongly?

Check the page source from that page. You won't find the text that you
copied. What I suspect is that the site owner has deliberately put
roadblocks in the way to prevent people from copying text as easily as
dragging a mouse across it.

In other words, the problem isn't only your odd choice of text editor,
it's also the pages that you choose to copy from. If you're going to
copy from a site like that, be prepared to do some post processing.

He who is Mayayana said on Mon, 9 Oct 2017 01:09:29 -0400:

The main problem you're having is
that you don't understand the distinctions.

I don't at all disagree. All this talk of utf-8 and ascii and ANSI is
driving me nuts in that I don't have the background for understanding.

Fundamentally, I have a keyboard that has a fundamental set of characters
on it that are the same characters that I want to show up in my cut and
pastes from a web site into file that is saved with the *.txt extension in
Windows.

I don't study languages, but the Babylonians seemed to communicate
perfectly well with such an arrangement of a small set of 26 fundamental
characters.

Why can't we be "allowed" to do the same?

Specifically I don't need nor want the "dastardly" non-keyboard characters.
All I'm trying to do is get rid of them as efficiently as possible.

Text editor usually refers to plain text. No font options,
color, etc. But plain text can be of different encodings.

I think there should be a "keyboard encoding" which is simply the
fundamental set of characters that can be typed on a typical (US American)
keyboard!


If I were to ask a Babylonian, I'd bet you he'd agree, and he was the one
who designed the alphabet (or so I'm told) so he knew what he was doing!

NOTE: I just looked up to doublecheck who created the alphabet, only to
find it's more complex than I had imagined - so I'll just say Babylonians
did it (but it's not that simple):
https://en.wikipedia.org/wiki/History_of_the_alphabet

A text editor is used to write plain text that doesn't
need formatting. It then saves that text to file as a simple
stream of bytes. Typically each byte represents a character.
Sometimes 2 bytes represent a character.

If those characters are "keyboard characters" - that's all I want.

Vi/vim/gvim is not a good text editor because it's designed
to be a coding editor.

My fingers are permanently engrained with muscle memory because my first
text editor ever (decades ago) was "vi" so I'm most efficient in vi because
I don't use the mouse to edit text files and I never will because nobody
ever typed faster by removing their fingers from the keyboard to mess with
a mouse.

Besides the horrendous UI, it doesn't
seem to handle different encodings. That's why you're having
trouble. Anything you paste is rendered as ASCII, because Vi
is assuming that you're writing code.

GVIM failed.
Notepad failed.
MS Word failed.

I don't think there is extent any suggested program that I've tested that
converts the "dastardly" dotardly curly quotes to straight quotes.

Of course vi can do a search and replace.
I think that will have to be the solution.

I just have to figure out how to type the curly quote when it doesn't exist
on my keyboard.

Do you know what I type below for "x" to represent the curly quote?
%s/x/"/g

That sequence in vi-speak, is:
% = for the entire file
s/x = search for "x" (where I need to know how to define a curly quote)
/"/g = and replace it with a straight quote, for every instance found

If I just knew what to put there for "x" to represent the curly quote, I'd
be done.

I find it helps to understand some of how it all works
behind the scenes in order to make sense of issues
like the one you're having with "dastardly text".

I admit information overload. It's more than I can handle.
All I would ask, at this point, is one thing:

How do I tell vi to recognize a curly quote?

On Mon, 9 Oct 2017 06:10:25 +0000 (UTC), harry newton
wrote:

He who is Char Jackson said on Mon, 09 Oct 2017 00:22:09 -0500:

The reason I'm using GVIM on Windows is that vi allows for quick edits.

I tried to use Google to find "gvim quick edits" and came up empty. What
does GVIM do that other text editors don't do? What are quick edits? I
assume those two words don't have their obvious meaning, because every
text editor does that, so what is it?

I apologize. My words were not clear. Hence I caused unnecessary confusion.

I appreciate that you followed the implied meaning of my words, which makes
it my fault for not explaining that "vi" is all I ever wanted. GVIM just
came with the package when I searched for the best "vi" editor on Windows.

I never use the GUI in GVIM. I never use the mouse. Both hands are on the
keyboard, because all I do with GVIM is use the "vi" part of VIM.

So what I meant by "quick edits" was my own concoction of words to describe
that fact that I can cut and paste and copy and rename and delete and
reorder, etc., all using keyboard presses.

For example, I can delete a line by typing "dd" and I can yank a line by
typing "yy" and I can paste those deleted or yanked lines ten lines lower
by typing "p" after jumping down ten lines with "10j" such that the
self-concocted phrase "quick edits" means something like this sequence:

yy10jp == this is a "quick edit" yanking a line to place it 10 lines lower

The actual keyboard presses would be different, but can't every text
editor do all of those things? I would say yes.

Thank you for the explanation, by the way!

He who is Mike S said on Sun, 8 Oct 2017 23:01:14 -0700:

Can you make a Word doc available for d/l somewhere that has all of the
characters you want to replace with one 'bad' character, space,
replacement character pair per line?

Yes, I can. And will. And just did.
http://wetakepic.com/images/2017/10/09/sample.jpg

1. This web page has the fundamental problem-pair set you ask about:
https://practicaltypography.com/straight-and-curly-quotes.html

2. So I created this simple sample text out of that web page:
https://pastebin.com/BFRYNs1c

2. I copied that sample text paste to this MS Word document:
http://www.filedropper.com/fixquote

3. I then copied from MS Word to the vi text editor to create this file:
https://www.sendspace.com/file/bhgsl8

All suffered from the same malady of having dotard curly quotes instead of
keyboard quotes. http://wetakepic.com/images/2017/10/09/sample.jpg