Speed of USB flahdrives

Carlos E.R. wrote:
On 01/12/2019 03.48, micky wrote:
I copied 80 photos totalling 340 Megs
from my HDD to a Sandisk Ultra USB 3 64GB
and it took about 15 seconds, then copied it back
and it took about 15 seconds.

On the curiosity note, if you repeat the test in Linux you can do things
like disable the cache and thus measure the stick raw speed. Or write
using the cache, then tell the system to "eject" the stick - which
efectively empties the cache for the stick.

In Windows I guess the thing would be to use a purpose made speed
testing program that talks to the system to empty the cache at the end.

15 seconds looks pretty fast.

...


You can do that ?

Linux is a bitch now at buffering.
It's hard to bench anything and get
a trustworthy result. The results can
easily be on the high side... Been there
and fought the Tshirt.

Even the benchmarking tools, the tools
were *designed* to measure the caching,
because the caching is impressive, and
that's what they wanted to show you.

Getting unbuffered results is getting harder
and harder to do. (Who knows, maybe even a
dropcache isn't enough.)

Paul

On 12/1/2019 4:41 AM, Andy Burns wrote:
Paul wrote:

he only reason for going with a USB3 stick at 100MB/sec
reads, is to avoid some 20MB/sec USB2 stick.

I believe the only USB3 stick I own, is the32GB dual-ended USB-C/USB-A
one that I bought *because* of the dual connectors, I stripped the funky
sliding plastic bits off it, so it's just a bare metal case.

Copying a few GB of large files, it can manage 42MB/s for about a
minute, then it drops down to 14MB/s for the remainder, it gets pretty
toasty on writes.


I've never bought a USB3 thumb drive. I have a bunch of USB2 thumb
drives, and I never thought the extra speed of USB3 justified the cost
of replacing the USB2s. I mostly use the drives just for my daily backup
of Quicken data, and USB2 is fast enough for those relatively small
files (just over 2GB).

If the drives I have die, and need to be replaced, I'll of course
replace them with USB3s, but there's no rush to do that.

--
Ken

On 01/12/2019 17.33, Paul wrote:
Carlos E.R. wrote:
On 01/12/2019 03.48, micky wrote:
I copied 80 photos totalling 340 Megs Â*Â* from my HDD to a Sandisk
Ultra USB 3 64GB
Â*Â* and it took about 15 seconds, then copied it back
Â*Â* and it took about 15 seconds.

On the curiosity note, if you repeat the test in Linux you can do things
like disable the cache and thus measure the stick raw speed. Or write
using the cache, then tell the system to "eject" the stick - which
efectively empties the cache for the stick.

In Windows I guess the thing would be to use a purpose made speed
testing program that talks to the system to empty the cache at the end.

15 seconds looks pretty fast.

...


You can do that ?

Linux is a bitch now at buffering.
It's hard to bench anything and get
a trustworthy result. The results can
easily be on the high side... Been there
and fought the Tshirt.

Even the benchmarking tools, the tools
were *designed* to measure the caching,
because the caching is impressive, and
that's what they wanted to show you.

Getting unbuffered results is getting harder
and harder to do. (Who knows, maybe even a
dropcache isn't enough.)

Sure. Try with dd:

time dd if=bigfile of=/usbmount/ oflag=direct bs=16M


Or the traditional

hdparm -tT /dev/stick


Or a variant like:

time ( cp source dest ; umount stick )




--
Cheers, Carlos.

Carlos E.R. wrote:
On 01/12/2019 17.33, Paul wrote:
Carlos E.R. wrote:
On 01/12/2019 03.48, micky wrote:
I copied 80 photos totalling 340 Megs from my HDD to a Sandisk
Ultra USB 3 64GB
and it took about 15 seconds, then copied it back
and it took about 15 seconds.
On the curiosity note, if you repeat the test in Linux you can do things
like disable the cache and thus measure the stick raw speed. Or write
using the cache, then tell the system to "eject" the stick - which
efectively empties the cache for the stick.

In Windows I guess the thing would be to use a purpose made speed
testing program that talks to the system to empty the cache at the end.

15 seconds looks pretty fast.

...

You can do that ?

Linux is a bitch now at buffering.
It's hard to bench anything and get
a trustworthy result. The results can
easily be on the high side... Been there
and fought the Tshirt.

Even the benchmarking tools, the tools
were *designed* to measure the caching,
because the caching is impressive, and
that's what they wanted to show you.

Getting unbuffered results is getting harder
and harder to do. (Who knows, maybe even a
dropcache isn't enough.)

Sure. Try with dd:

time dd if=bigfile of=/usbmount/ oflag=direct bs=16M


Or the traditional

hdparm -tT /dev/stick


Or a variant like:

time ( cp source dest ; umount stick )

The hdparm one... may not be testing what you think
it is testing. It took me the longest time to figure
that out, because there were so many wrong references.

Maybe the "dd" would work, but you really need to
test these things to see if the results are reasonable.

I'm curious if you've spotted and characterized the
Linux system write cache yet. It's bananas. The
output side of the cache doesn't write a thing,
until the cache hits a certain level of fill.
That's wasted time, that could have been used
to receive writes. I couldn't believe this, when
I first noticed it happening.

System write caches (now present on both Windows
and Linux), represents a danger on large-RAM machines,
due to the possibility a user will delete a source
file (after it "appears" the transfer is complete),
then the machine loses AC power and the file is lost
and fragments removed by journaling. You can lose a
file entirely, if you're not careful, and the machine
has a lot of RAM.

Paul

Paul wrote:

Andy Burns wrote:

Copying a few GB of large files, it can manage 42MB/s for about a
minute, then it drops down to 14MB/s for the remainder

They do make some better USB3 sticks now. In terms of
nominal transfer rates.

I found the spec sheet for mine, it only claims 150MB/s read (which it
does achieve) and "write speeds lower" which I guess is accurate ...

On 01/12/2019 20.33, Paul wrote:
Carlos E.R. wrote:
On 01/12/2019 17.33, Paul wrote:
Carlos E.R. wrote:
On 01/12/2019 03.48, micky wrote:
I copied 80 photos totalling 340 MegsÂ*Â*Â* from my HDD to a Sandisk
Ultra USB 3 64GB
Â*Â* and it took about 15 seconds, then copied it back
Â*Â* and it took about 15 seconds.
On the curiosity note, if you repeat the test in Linux you can do
things
like disable the cache and thus measure the stick raw speed. Or write
using the cache, then tell the system to "eject" the stick - which
efectively empties the cache for the stick.

In Windows I guess the thing would be to use a purpose made speed
testing program that talks to the system to empty the cache at the end.

15 seconds looks pretty fast.

...

You can do that ?

Linux is a bitch now at buffering.
It's hard to bench anything and get
a trustworthy result. The results can
easily be on the high side... Been there
and fought the Tshirt.

Even the benchmarking tools, the tools
were *designed* to measure the caching,
because the caching is impressive, and
that's what they wanted to show you.

Getting unbuffered results is getting harder
and harder to do. (Who knows, maybe even a
dropcache isn't enough.)

Sure. Try with dd:

time dd if=bigfile of=/usbmount/ oflag=direct bs=16M


Or the traditional

hdparm -tT /dev/stick


Or a variant like:

Â* time ( cp source dest ;Â* umount stick )

The hdparm one... may not be testing what you think
it is testing. It took me the longest time to figure
that out, because there were so many wrong references.

I have not tested it with sticks, that's true. Only with hard disks.


Maybe the "dd" would work, but you really need to
test these things to see if the results are reasonable.

Oh, I have done that :-)

It is the command line I use to "burn" installation isos to sticks, and
is faster than others.

I have also a little "menu command" written on "mc" to do the copy using
that dd command - when I want to copy multiple (and many) files to a stick.

I'm curious if you've spotted and characterized the
Linux system write cache yet. It's bananas. The
output side of the cache doesn't write a thing,
until the cache hits a certain level of fill.
That's wasted time, that could have been used
to receive writes. I couldn't believe this, when
I first noticed it happening.

No, I have not bothered. It is quite complex. And can change with kernel
version. You can try to write to a stick several gigabytes (say, a DVD
image), and while you have RAM available it blazes away - till all ram
is spent and then it crawls at the real write speed of the stick. And
this using of the entire ram causes the entire system to suffer, because
there is little cache or buffers for other processes.


System write caches (now present on both Windows
and Linux), represents a danger on large-RAM machines,
due to the possibility a user will delete a source
file (after it "appears" the transfer is complete),
then the machine loses AC power and the file is lost
and fragments removed by journaling. You can lose a
file entirely, if you're not careful, and the machine
has a lot of RAM.

Yes, it is certainly possible. You have to umount the target, or use a
command to flush caches.

People should be strongly reminded that computers hate power failures
and that they really need to use UPS. Or laptops.

With certain files, I'm paranoid and do a file compare between source
and target files.

--
Cheers, Carlos.

In alt.comp.os.windows-10, on Sat, 30 Nov 2019 21:48:33 -0500, micky
wrote:



A friend borrowed my spare camera for her grandson's wedding last April
and took pictures. At her 3 sons' weddings, even though there was a
hired photographer, and iirc a videographer at each one, she took her
own pictures also and I know she would take them to get developed the
next day, probably same-day printing. But this time, her battery was
dead and they didnt' have the right charger for where we were.

So I lent her mine, but when I got back from my vacation, I couldn't
find my main camera. I debated whether to call the rentacar company,
but was sure they didn't have it. So basically, I took my anger at
myself out on my friend. Shame on me.

I'd forgotten that a lot of pictures from the wedding *I* had taken were
in the camera, so I can give both hers and mine to her at the same time,
on the same CD/flashdrive. Plus videos. Things she didn't see, let
alone film

And I'm giving her copies of FSViewer, FinePix Viewer, and Irfanview, in
case she doesn't know about all of these. I wonder what she does use.

So unless she's gotten a lot sicker, God forbid, I didn't hurt her with
the delay.

I myself don't always look at my pictures until years after I take them.
Up until then I can remember what I saw.

Finally after 6 months, I found
the camera, in the computer bag where I'd looked twice before. It was at
the bottom of one of the main pockets, below a fold (not supposed to be
a fold but it's got one anyhow.)

Now that I have my camera, I'm diligently working to send her her
pictures, and some of mine. She's not that technical, I think, and is
having her 3rd major disease since 2002 (Lymphoma, kidney transplant
(surgery 2 to 3 years ago, from her son), and breast cancer surgery) so
I don't want to send her an especially slow flashdrive . 80 of her
pictures (of her grandon's wedding) and maybe 100's of mine of related
subjects. So if I don't have one, I'll buy** a USB3 drive for her so if
she wants to copy the other pictures (mine) that I'm including, it won't
take her hours.

**I should go shopping tomorrow, but it's madness Sunday and I hate
that.

And I know too now that for photos, usb2 can be pretty slow. (When I
backed them up from the PC to the USB backup HDD, it took about 30
minutes for over a 1000 photos. About a second per photo or less most of
the time.)

One thing I discovered, is you have to set your Power
control panel to "High Performance" if testing USB sticks.
That takes some of the inconsistency out of the
graph on the screen. At least that gives me closer
to a "flat line" like it's supposed to be.

Paul

Is there a "miklitary grade" type that one could buy with complete
confidence?

Even though they're twice the price?

On 2019-12-01 6:46 p.m., Peter Jason wrote:

One thing I discovered, is you have to set your Power
control panel to "High Performance" if testing USB sticks.
That takes some of the inconsistency out of the
graph on the screen. At least that gives me closer
to a "flat line" like it's supposed to be.

Paul

Is there a "miklitary grade" type that one could buy with complete
confidence?

Even though they're twice the price?


I can't answer but I can tell you my experience with quality and honest
specs.
I own 7 Adata products and all of them have met or exceeded their
advertised specs and have been totally trouble free.

No I am not associated with the company, Just a very satisfied user.

2 x NVME drives 512 GB
2 x SSD drives 256 GB
3 x USB drives 32 GB

Rene

Peter Jason wrote:
One thing I discovered, is you have to set your Power
control panel to "High Performance" if testing USB sticks.
That takes some of the inconsistency out of the
graph on the screen. At least that gives me closer
to a "flat line" like it's supposed to be.

Paul

Is there a "miklitary grade" type that one could buy with complete
confidence?

Even though they're twice the price?

If you could find someone to make you an SLC
one, then yes, for twice the price you get
a stick that lasts for a longer period of time.

However, who can you trust ?

That's the problem. You won't know what is
in there, until you get it home and disassemble
it and check.

For example, an advert I posted here a while ago,
the dude shows pictures of the USB stick flash design,
including a chip "NW711", which is (apparently) a
Micron SLC flash chip in BGA package.

The problem is, the advert said they were "sold out"
(the model in the picture), and some stick designs
which didn't have pictures, were for sale instead.

The last time a company (a "brand" invented for the purpose)
sold SLC, the adverts lasted for maybe four months
and they disappeared. There were no pictures of the
guts of those, and I assumed based on capacity value,
that these were TLC ones. I didn't know anyone was
making actual useful sizes of SLC chips. And it's
my suspicion Micron won't be making them for much
longer either. Because the web information seems
to be poorly maintained, and maybe they're just selling
off remaining stock.

I found a smaller company (specialty company) making
SLC flash chips, but the chip capacities are quite small.

Sometimes you can find these sticks on Mouser or Digikey
or the like. Again, a specialty company makes the
sticks, and they're probably 4x or 5x the price of
the Amazon ones. The sticks might be 2GB or 4GB,
in other words, not keeping pace with the factor
of four capacity difference that should exist.
(If Walmart can sell a 32GB TLC stick, then a 8GB SLC
should be available too at Mouser.) Well, the Micron ones
are going to be using multiple silicon dies inside,
and they could be stacked. Then you have more
capacity options. 3DNand, on the other hand, the
strings of cells are oriented vertically in
the silicon. And the X-Y shape of the cells is
likely to be different than in a planar NAND.
(Then you stack planar dies to make higher
capacity products.)

Planar flash silicon dies are shaved down to make them
thin. They do that to encourage stuff like
stacking (without making the chip package look "bloated").

It's pretty amazing how they make the things.
The part that sucks, is how they're binned and
sold. Hardly anything goes into the garbage,
which means we need a means to sell off the
"error-filled" ones, and a USB TLC stick would
seem to be an excellent disposal method :-)
I wouldn't mind these practices nearly so much,
if I thought the controller chip in the USB
flash stick, was doing an excellent job of
"managing" its partner Flash chip. I'm not
convinced those controller chips have a full
feature set. The sticks seem to die too easily.

If you had SLC flash chips in the stick, the
controller could be as dumb as dirt, and you'd
still get years of usage from the stick.

Paul