slow file access in large folder

I have a folder on a NTFS drive with about 2500 files in it and when I
browse the files in the folder with windows explorer, it seems like it takes
a while to load all of the files into memory. I never had this slowdown in
win2000. At first, I thought it was because of accessing the files over the
network, but then I moved them to my harddrive and still had the same
performance.
My first suspicion is that XP is slow because it has to read all of the file
attributes like security settings. Is this correct? Why didn't this happen
in Win2000? Is XP doing something different to the files than Win2000? Is
there a way to disable it?
This also happens when I browse through programs like Word or Solidworks.
So it's not just windows explorer.
I've read a couple of newsgroups where some people talk about increasing the
"folder cache" size. Would this help? How do you do it?
Also, would upgrading my hardware help? This is what I have:
2.0ghz P4
512mb ram
40gig 7200rpm HD

BTW, I did defrag.