is "Everything" doing some mining?

In message , VanguardLH
writes:
[]
Its immediate search results require previous searching. Like Windows
own indexing, Search Everything has to build its own index. I suspect
it occasionally performs an update by scanning the drives to detect
anything that changed that its service might've not captured at the time
of change.

I assume the some tens of seconds Everything takes when I start it
(sometimes; I haven't figured when. Possibly only first time after a
reboot?) is it doing an index. I haven't really looked at what CPU it's
using during that time, or timed how long it takes - but, even though it
might actually be over a minute, it does end. Whereas this 24/25% CPU
seems _not_ to. (Obviously I haven't - consciously - let it run for
hours, though I think I've sometimes come back to the machine and found
it happening.) So if it can do a full index in a minute or so, then
settle down, what exactly is it _doing_ when it takes off like that?

Unless I missed it, I did not see an option of when to reindex in the
background.

Since I stopped (because it had gone into 25% CPU mode) and restarted it
near the beginning of this thread, it has now been running for over 20
hours, and is still behaving itself - 00 in task manager - although it
might have had a busy session while I was asleep. [I don't shut the
machine down.]
[]
If you are concerned about data mining, use a network monitor, like
wireshark, to check if the program is phoning home. Make sure to
disable its auto-update check to eliminate that network traffic. If you

I'm not - I really don't think voidtools are either mining bitcoins
(which I _think_ I wouldn't mind anyway) or data mining. It was just a
quick and not really thought-out reaction to seeing CPU usage suddenly
not being at idle level.
[]
DOES anyone else experience this sudden rise in CPU usage of Everything,
or - like several other funnies! - is it just me?
--
J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)[email protected]+H+Sh0!:`)DNAf

The thing about smut is it harms no one and it's rarely cruel. Besides, it's a
gleeful rejection of the dreary and the "correct".
- Alison Graham, RT 2014/10/25-31

"J. P. Gilliver (John)" wrote

| I had never thought of Explorer as a search program. Probably because
| (to me anyway) it's UI (for that function) is very non-intuitive.

That's why I said "modified Explorer". Explorer is
a GUI presentation of the file system. It's just
a fancy listing with icons. Everything seems to be
the same basic idea, but with search added.

So while most search tools do their thing and
show you a list, Everything shows you a GUI
representation.

Unless something has changed, Windows search is
actually an Explorer Bar -- a shell extension window
that docks on the left side of Explorer windows. But
I don't think I've seen that ever since that obnoxious
puppy cartoon when I first tried XP.

J. P. Gilliver (John) wrote:
In message , VanguardLH
writes:
[]
Its immediate search results require previous searching. Like Windows
own indexing, Search Everything has to build its own index. I suspect
it occasionally performs an update by scanning the drives to detect
anything that changed that its service might've not captured at the time
of change.

I assume the some tens of seconds Everything takes when I start it
(sometimes; I haven't figured when. Possibly only first time after a
reboot?) is it doing an index. I haven't really looked at what CPU it's
using during that time, or timed how long it takes - but, even though it
might actually be over a minute, it does end. Whereas this 24/25% CPU
seems _not_ to. (Obviously I haven't - consciously - let it run for
hours, though I think I've sometimes come back to the machine and found
it happening.) So if it can do a full index in a minute or so, then
settle down, what exactly is it _doing_ when it takes off like that?

Unless I missed it, I did not see an option of when to reindex in the
background.

Since I stopped (because it had gone into 25% CPU mode) and restarted it
near the beginning of this thread, it has now been running for over 20
hours, and is still behaving itself - 00 in task manager - although it
might have had a busy session while I was asleep. [I don't shut the
machine down.]
[]
If you are concerned about data mining, use a network monitor, like
wireshark, to check if the program is phoning home. Make sure to
disable its auto-update check to eliminate that network traffic. If you

I'm not - I really don't think voidtools are either mining bitcoins
(which I _think_ I wouldn't mind anyway) or data mining. It was just a
quick and not really thought-out reaction to seeing CPU usage suddenly
not being at idle level.
[]
DOES anyone else experience this sudden rise in CPU usage of Everything,
or - like several other funnies! - is it just me?

You could try running Sysinternals Process Monitor, and see if the repetitive
loop it gets into, emits any system calls. And then you might get an idea
of what to look at.

Paul

On Sat, 16 Mar 2019 18:59:20 -0400, Paul wrote:

"Everything.exe" indexes in two stages, if it is starting
from scratch.

It reads the $MFT and parses it. This gives a list
of file names, but not their size or creation date.
This might take two seconds. The voidtools designers
really should have stopped at this point.

Fortunately, cooler heads prevailed and they didn't stop at that point,
probably recognizing that a search tool with such a limited set of
capabilities would be essentially worthless.

It's the second phase which is more expensive. If they
want to add dates and sizes and so on, file details,
that requires "walking the tree".

Finally, at some point, the "list" needs sorting.
Perhaps as a means of reducing search time on actual
searches (binary probing?).

None of these activities should particularly leave
the CPU railed, forever... Sorting a list takes
time. But not infinite time.


--

Char Jackson

In message , Paul
writes:
[]
You could try running Sysinternals Process Monitor, and see if the repetitive
loop it gets into, emits any system calls. And then you might get an idea
of what to look at.

Paul

All these things I have to remember, next time it does it! (It's now
been running for over 24 hours and hasn't done it, unless it did it _and
stopped doing it_ while I was asleep.)

Has anyone else had the odd behaviour - does an index when you start it
(I think), then settles down to idle, then does something else that
takes longer than the initial index (or whatever) activity? Or is it
just me?
--
J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)[email protected]+H+Sh0!:`)DNAf

"There are a great many people in the country today who, through no fault of
their own, are sane." - Monty Python's Flying Circus

On Sat, 16 Mar 2019 18:04:21 -0700, "James Davis"
wrote:

I use Search "Everything" Version 1.4.1.877 (x64) which I
downloaded about 6 months ago to update a much earlier version.
It has an "Advanced Search" that will search content and many
more things. --We should all get the latest version [;-)].

The latest version at https://www.voidtools.com/ appears to be
1.4.1.935.

But
lately, I too have been noticing Search "Everything" interfering
with Windows 7 Explorer after using MSIE-11 while Search
"Everything" is active.

I haven't noticed any similar issues, but then, I don't use IE11 (or any
version of IE).

--

Char Jackson

"Paul" wrote

| "Everything.exe" indexes in two stages, if it is starting
| from scratch.
|
| It reads the $MFT and parses it. This gives a list
| of file names, but not their size or creation date.
| This might take two seconds. The voidtools designers
| really should have stopped at this point.
|

According to their FAQ you can eliminate those other
steps in the settings. So people who just want to find
"My 2017 taxes.txt" can search for that while Char
searches for *only* the copy of "My 2017 taxes.txt" that
has a last modified date later than 2016.

On Sat, 16 Mar 2019 20:27:39 -0600, "Bill in Co"
[email protected] wrote:

Mayayana wrote:
"Bill in Co" [email protected] wrote

I use an older version of FileLocator Pro, the big brother to Agent
Ransack, which has a lot more options, like excluding directories from
searches, which I find very advantageous. But it's not free, and
unfortunately, has gotten a bit pricey over the years. But the option
to exclude directories from searches greatly speeds up finding things,
especially since I'm not using any indexing, by choice.


AR lets you start at any level/location. Maybe
you mean doing something like excluding System32
when searching Windows?

Exactly. And even more than that!


Search Everything lets you exclude specific files, specific folders,
hidden files and folders, and/or system files and folders.

--

Char Jackson

On 03/17/2019 12:35 PM, Char Jackson wrote:
On Sat, 16 Mar 2019 20:27:39 -0600, "Bill in Co"
[email protected] wrote:

Mayayana wrote:
"Bill in Co" [email protected] wrote

I use an older version of FileLocator Pro, the big brother to Agent
Ransack, which has a lot more options, like excluding directories from
searches, which I find very advantageous. But it's not free, and
unfortunately, has gotten a bit pricey over the years. But the option
to exclude directories from searches greatly speeds up finding things,
especially since I'm not using any indexing, by choice.


AR lets you start at any level/location. Maybe
you mean doing something like excluding System32
when searching Windows?

Exactly. And even more than that!


Search Everything lets you exclude specific files, specific folders,
hidden files and folders, and/or system files and folders.


Both excellent programs, Each is better for different job, wouldn't be
without them. :-)

Rene

Char Jackson wrote:
On Sat, 16 Mar 2019 20:27:39 -0600, "Bill in Co"
[email protected] wrote:

Mayayana wrote:
"Bill in Co" [email protected] wrote

I use an older version of FileLocator Pro, the big brother to Agent
Ransack, which has a lot more options, like excluding directories from
searches, which I find very advantageous. But it's not free, and
unfortunately, has gotten a bit pricey over the years. But the option
to exclude directories from searches greatly speeds up finding things,
especially since I'm not using any indexing, by choice.


AR lets you start at any level/location. Maybe
you mean doing something like excluding System32
when searching Windows?

Exactly. And even more than that!


Search Everything lets you exclude specific files, specific folders,
hidden files and folders, and/or system files and folders.

But I'm assuming requires indexing. Thanks, but no thanks, for reasons
previously enumerated. :-)

J. P. Gilliver (John) wrote:

Has anyone else had the odd behaviour - does an index when you start it
(I think), then settles down to idle, then does something else that
takes longer than the initial index (or whatever) activity? Or is it
just me?

Of the 8 respondents, so far, none have noted their own encounter with
the sporadic jump to 25% CPU usage just for the everything.exe process.
Could be just you. Could be there is a jump but it is so short-lived
that users wouldn't notice it versus for you where it sticks at the 25%
CPU usage for long enough for you to notice.

Are you using anything regarding the CPU, like affinity, on how to run
the program? Or some CPU dispatcher or priority enhancer (e.g., Process
Tamer, Bill2's Process Manager, Prio, Process Lasso) instead of using
just the one in Windows? As I recall, some of those could also define
an core affinity for some processes or how many cores a process can use.

Do you run any process (manually, startup, scheduled) that results in
renaming or deleting entire folders? Doing such is expensive to
Everything in that the entire folder and all its files, especially if
many, have to update its index database. Perhaps you are dumping tons
of files into, say, the %temp% folder and then run a cleanup tool either
automatically due to thresholds (number of files, aggregate size),
manually, or scheduled. When there are huge changes to the file system,
Everything has a lot of work to do. It's not just folder deletes that
could impact Everything. For example, if you run SQL Server Management
Studio (SSMS) which massively interacts with its drive by frequently
writing & deleting lots of session data, Everything has to keep catching
up. When Everything's CPU usage jumps up, you might want to look at
your disk activity to see what other processes might be doing lots of
changes in the file system.

Everything uses multiple threads to complete much of its work in
parallel, and that could impact CPU usage. You can edit the
everything.ini file to change the max_threads. I think max_threads=0
means to disable throttling of thread count. You could max_threads=x,
where x is the number of logical CPUs (cores) minus 1, or to a lesser
number to generate less threads for Everything. With less threads,
updating will take longer but as less CPU usage.

https://www.voidtools.com/support/everything/ini/

I don't recall you mentioning what file system you are using. Most of
us probably assumed you are using NTFS, but maybe you are using FAT.
Everything can use the Journaling in NTFS to detect when a file has
changed. In Everything for the NTFS config, is "Enable USN Journal"
enabled (assuming you are using NTFS for the file system)?

On 03/17/2019 6:06 PM, VanguardLH wrote:
J. P. Gilliver (John) wrote:

Has anyone else had the odd behaviour - does an index when you start it
(I think), then settles down to idle, then does something else that
takes longer than the initial index (or whatever) activity? Or is it
just me?

Of the 8 respondents, so far, none have noted their own encounter with
the sporadic jump to 25% CPU usage just for the everything.exe process.
Could be just you. Could be there is a jump but it is so short-lived
that users wouldn't notice it versus for you where it sticks at the 25%
CPU usage for long enough for you to notice.

Are you using anything regarding the CPU, like affinity, on how to run
the program? Or some CPU dispatcher or priority enhancer (e.g., Process
Tamer, Bill2's Process Manager, Prio, Process Lasso) instead of using
just the one in Windows? As I recall, some of those could also define
an core affinity for some processes or how many cores a process can use.

Do you run any process (manually, startup, scheduled) that results in
renaming or deleting entire folders? Doing such is expensive to
Everything in that the entire folder and all its files, especially if
many, have to update its index database. Perhaps you are dumping tons
of files into, say, the %temp% folder and then run a cleanup tool either
automatically due to thresholds (number of files, aggregate size),
manually, or scheduled. When there are huge changes to the file system,
Everything has a lot of work to do. It's not just folder deletes that
could impact Everything. For example, if you run SQL Server Management
Studio (SSMS) which massively interacts with its drive by frequently
writing & deleting lots of session data, Everything has to keep catching
up. When Everything's CPU usage jumps up, you might want to look at
your disk activity to see what other processes might be doing lots of
changes in the file system.

Everything uses multiple threads to complete much of its work in
parallel, and that could impact CPU usage. You can edit the
everything.ini file to change the max_threads. I think max_threads=0
means to disable throttling of thread count. You could max_threads=x,
where x is the number of logical CPUs (cores) minus 1, or to a lesser
number to generate less threads for Everything. With less threads,
updating will take longer but as less CPU usage.

https://www.voidtools.com/support/everything/ini/

I don't recall you mentioning what file system you are using. Most of
us probably assumed you are using NTFS, but maybe you are using FAT.
Everything can use the Journaling in NTFS to detect when a file has
changed. In Everything for the NTFS config, is "Enable USN Journal"
enabled (assuming you are using NTFS for the file system)?



Just did another test on mine watching task manager all the while,
Opened Search everything and typed in one word, usage was at 0% and
jumped to 2% for 2 or 3 seconds then returned to 0%, CPU dropped back to
1.61 GBs. which is about half speed,
and idled there for about 10 minutes as I watched. memoru usage was
steady at 2.2 GB
So in my case at least the program is running normal.

Rene

Char Jackson wrote:
On Sat, 16 Mar 2019 18:04:21 -0700, "James Davis"
wrote:

I use Search "Everything" Version 1.4.1.877 (x64) which I
downloaded about 6 months ago to update a much earlier version.
It has an "Advanced Search" that will search content and many
more things. --We should all get the latest version [;-)].

The latest version at https://www.voidtools.com/ appears to be
1.4.1.935.

But
lately, I too have been noticing Search "Everything" interfering
with Windows 7 Explorer after using MSIE-11 while Search
"Everything" is active.

I haven't noticed any similar issues, but then, I don't use IE11 (or any
version of IE).


I wonder. If you had a million files in an IE cache,
would the dates on the files change when you use IE ?

I would want to understand just how many files were
on this machine, for a starter. To see if there is
"any potential way for scaling to account for it".

Like, if there were 10 million files, you would
expect the indexing (if and when done) to take
longer than an average install/use of 0.5 million files.

My brother (in his researcher era), contacted me once,
talking about slow system performance. And happened
to mention he had IE cache cranked to the moon. Back
then he was collecting scientific papers in his field
for usage in his company database, so he was using
the web browser 8 hours a day at the time. And it turned
out, he had an inordinate amount of files in that
cache. Turning the cache down, returned his laptop
to "normal".

So when a person mentions IE, I hark back to that
datapoint, of setting the IE cache a little too high,
and ending up with too many files sloshing around in there.

Paul

In message , Rene Lamontagne
writes:
On 03/17/2019 6:06 PM, VanguardLH wrote:
J. P. Gilliver (John) wrote:

Has anyone else had the odd behaviour - does an index when you start
it
(I think), then settles down to idle, then does something else that
takes longer than the initial index (or whatever) activity? Or is it
just me?
Of the 8 respondents, so far, none have noted their own encounter
with
the sporadic jump to 25% CPU usage just for the everything.exe process.
Could be just you. Could be there is a jump but it is so short-lived
that users wouldn't notice it versus for you where it sticks at the 25%
CPU usage for long enough for you to notice.

Yes, for me I notice it has gone to 25% (of an apparently 4-core
processor), and doesn't come back down (even if I close the GUI).

Are you using anything regarding the CPU, like affinity, on how to
run
the program? Or some CPU dispatcher or priority enhancer (e.g., Process
Tamer, Bill2's Process Manager, Prio, Process Lasso) instead of using
just the one in Windows? As I recall, some of those could also define
an core affinity for some processes or how many cores a process can use.

I've never even heard of any of those other than Process Lasso, and am
not running that (-:.

Do you run any process (manually, startup, scheduled) that results
in
renaming or deleting entire folders? Doing such is expensive to

Nothing I can think of.

Everything in that the entire folder and all its files, especially if
many, have to update its index database. Perhaps you are dumping tons
of files into, say, the %temp% folder and then run a cleanup tool either
automatically due to thresholds (number of files, aggregate size),
manually, or scheduled. When there are huge changes to the file system,
Everything has a lot of work to do. It's not just folder deletes that
could impact Everything. For example, if you run SQL Server Management
Studio (SSMS) which massively interacts with its drive by frequently
writing & deleting lots of session data, Everything has to keep catching

I don't think I'm doing anything like that! The main prog.s I am
normally running are an old Firefox and a new Chrome, my news/email
client, Notepad+, Everything, Brother's Keeper - genealogy, a biggish
database handler but not actually doing anything most of the time), and
the odd tiddler like AllChars, noisy keyboard, and the like. Nothing big
- I'm no power user! About the only CPU-intensive thing I ever run is
very occasional video conversion/editing, but I know when I'm doing
that, and _expect_ the fan to take off.

up. When Everything's CPU usage jumps up, you might want to look at
your disk activity to see what other processes might be doing lots of
changes in the file system.

I'll look at the disk light if it ever does it again. (It's now been
running thirtysomething hours in "placid" mode.)

Everything uses multiple threads to complete much of its work in
parallel, and that could impact CPU usage. You can edit the
everything.ini file to change the max_threads. I think max_threads=0
means to disable throttling of thread count. You could max_threads=x,
where x is the number of logical CPUs (cores) minus 1, or to a lesser
number to generate less threads for Everything. With less threads,
updating will take longer but as less CPU usage.

It's set to 0 at the moment. When I've looked at task manager when it
goes to 25%, it isn't all in one core, though it's not symmetrical - two
of them seem busier than the other two.

https://www.voidtools.com/support/everything/ini/
I don't recall you mentioning what file system you are using. Most
of
us probably assumed you are using NTFS, but maybe you are using FAT.
Everything can use the Journaling in NTFS to detect when a file has
changed. In Everything for the NTFS config, is "Enable USN Journal"
enabled (assuming you are using NTFS for the file system)?

NTFS. I did on another machine try using FAT, but I found Everything
worked a lot less well with it: I don't remember in what way, but it was
and has been the only thing that's ever made me use NTFS. (I've used
NTFS when a machine has come set up with it. I can't remember how I came
to be using FAT on that machine - maybe I was setting up a new partition
or something.)


Just did another test on mine watching task manager all the while,
Opened Search everything and typed in one word, usage was at 0% and
jumped to 2% for 2 or 3 seconds then returned to 0%, CPU dropped back
to 1.61 GBs. which is about half speed,
and idled there for about 10 minutes as I watched. memoru usage was
steady at 2.2 GB
So in my case at least the program is running normal.

Rene

Thanks. Yes, I've just tried the same - as I typed characters into the
search box, Everything.exe in Task Manager went to 03 for one Task
manager refresh interval (I seem to have Task Manager set to Update
Speed - Normal, which seems to be about once a second), then drop back
to 00, for each character I typed.
--
J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)[email protected]+H+Sh0!:`)DNAf

"I hate the guys that criticize the enterprise of other guys whose enterprise
has made them rise above the guys who criticize!" (W9BRD, former editor of
"How's DX?" column in "QST")

Bill in Co wrote:
Char Jackson wrote:
On Sat, 16 Mar 2019 20:27:39 -0600, "Bill in Co"
[email protected] wrote:

Mayayana wrote:
"Bill in Co" [email protected] wrote

I use an older version of FileLocator Pro, the big brother to Agent
Ransack, which has a lot more options, like excluding directories from
searches, which I find very advantageous. But it's not free, and
unfortunately, has gotten a bit pricey over the years. But the option
to exclude directories from searches greatly speeds up finding things,
especially since I'm not using any indexing, by choice.

AR lets you start at any level/location. Maybe
you mean doing something like excluding System32
when searching Windows?
Exactly. And even more than that!

Search Everything lets you exclude specific files, specific folders,
hidden files and folders, and/or system files and folders.

But I'm assuming requires indexing. Thanks, but no thanks, for reasons
previously enumerated. :-)

Turning off indexing, doesn't speed anything up.

Indexing works fine on NTFS for these reasons:

1) A once-daily enumeration by an indexer, builds a
reliably list of items for that day. Leaving too long
between "full" indexes means an index could be corrupted
or out-of-sync. On Windows 10, Microsoft believes this
interval to be 3 months (90 days). That's how often the
Indexing feature on Win10 re-does the entire index. On
a moderate sized volume, a content index can take 3 hours.

2) The NTFS USN Journal notes both the creation and
deletion of files. I could have eight programs running
at the same time, each one creates a file... None of the
"events" get lost. The USN Journal accurately records all
eight of them. The various search programs (more than one
program if you want), keep a read pointer to the USN Journal,
and like a laundry list, they read the list and make note
of the current "point-in-time", then add those eight files
to their list (by Insertion Sort perhaps). *Nothing*
gets lost on NTFS. You can even read the USN Journal
your own self, and in plain text, see what it logs.

The Everything.exe search of the index file uses
binary probing of a sorted list. It splits the list
in two. It splits the list in four. And so on,
until the exact "fraction" of the list, with only
one entry, is located. If there are 1048576 files,
it takes 20 probes. If there are 16777216 files, it
takes 24 probes. It's log base 2. The probes themselves
are file reads. Even pessimistically, I could probably
do 40-50 of those a second, so a search could take
half a second on a largish index file. With read ahead
and track caching, the probes are likely to be significantly
faster than that, on a modern HDD with large DRAM cache.

The sequential indexing by programs such as Agent Ransack
(which uses no index), are not even remotely close to
the half second on a bad day that it's going to take
Everything.exe with one of its prepared lists.

The thing Everything.exe cannot help with, is FAT32 volumes.
No USN journal. The FAT32 volumes need more frequent
indexing if you don't want to "lose anything" concerning
the thoroughness of the indexing process. But NTFS is
well set up to aid this sort of indexing activity.

You could make an argument, that the content indexing
Windows 10 does, adds "noise" to the search list output,
and I could agree with that. But with a little care
in use of the search language, that is easily
controlled. For example "ext:jpg" ensures you
see only JPG files. And you can also say stuff
like "filename:fluffy.jpg", and prevent content
with the word "fluffy" from accidentally appearing
in the list. On the old Windows XP search, the usage
of separate boxes means "less language" need be
learned with regard to trimming that sort of noise.

Paul