Spotlight index takes over a thousand hours on Leopard

Hi,

I installed an app that should remain nameless. It had a button to re-index spotlight which I regrettably clicked. After about an hour I looked at the progress. The bar was around in the middle, but it said something like 8500 hours remaining. Within minutes it was up to over 10,000 hours remaining.

I've done quite a bit of research and it seems like this happens when there is a corrupt file somewhere. I have no idea how to find this corrupt file. Any idea?

While doing the research, I've found instructions to clear the spotlight index and try again. Specifically this is what I've tried:

mdutil -i off /
mdutil -i off /Volumes/Backups
rm -rf /.Spotlight*
rm -rf /Volumes/Backups/.Spotlight*
mdutil -E /
mdutil -E /Volumes/Backups
mdutil -i on /
mdutil -i on /Volumes/Backups

It still estimates over a thousand hours.

I've tried turning one volume on at a time and letting spotlight try to index just the one volume. Both estimate very long hours still. Makes me wonder if I have corrupt files in both volumes or if there is something else.

I have tried adding several folders to the spotlight privacy hoping to exclude the corrupt file, but it doesn't seem to help.

Any suggestions?

Thanks.

Have you tried Maintenance? You can rebuild the Spotlight index in there...

Download Maintenance for Mac - OS X maintenance utility. MacUpdate.com

No I have not. Will try it right away (the current rebuilding had almost 46,000 hours left).

Maintenance requires Snow Leopard.

I will add my entire home folder into the spotlight privacy now and see if that works.

Name and shame otherwise others may fall into the same bottomless pit.

I don't believe that the program was the cause of the problem. That is why I did not mention it. I actually think its a great program. At this point, I have wiped out the spotlight index and attempted to re-create it from scratch. The program is therefore no longer a part of the process.

The problem is spotlight's indexing. And possibly it's inability to skip over corrupt files.

It worked when I put my entire home folder inside the spotlight privacy. So the problem must be a corrupt file somewhere in there.

Any ideas on how I can find the culprit?

Is there some mac command line tool that checks a file? I could then easily use find with that command to test all files in my home folder.

Thanks.

I'm going to be honest, it's just weird you won't name the program, especially when you seemed to initially imply it was the source of the issue.

helpmhost said:

It worked when I put my entire home folder inside the spotlight privacy. So the problem must be a corrupt file somewhere in there.

Any ideas on how I can find the culprit?

Is there some mac command line tool that checks a file? I could then easily use find with that command to test all files in my home folder.

Thanks.

Click to expand...

May have just been a corrupt index file. Remove your home directory and see if it'll index correctly.

Did you really just tell the guy to delete his home directory? I hope he didn't try to do that... if he did/does... *chuckle* Could try a new user account, then move your data from one to the other, THEN delete the old home directory...

Use Onyx to rebuild spotlight index. If that doesn't work, well, you might want to check the health of your hard disk.

randalthor said:

Did you really just tell the guy to delete his home directory? I hope he didn't try to do that...

Click to expand...

No, if you read the thread, I telling him to remove it from the spotlight exclusion list that we told him to add it to earlier.

Give me some credit, I've been the Community Leader here since 2003, and using Macs much longer than that. I know what I'm talking about.

It does not work when my home folder is not in the spotlight privacy.

I am going to try Onyx. (I also found the Leopard version of Maintenance, will try that too).

Admittedly, I hadn't read the whole thread, haha. I was just thinking, "That can't be right!"

Onyx and Maintenance only start an indexing. I am already able to do that via the command line. I was hoping they might have done something more.

I've removed my home folder from the privacy and added some folder inside it and it was doing well until it got to 85% and then the hours start to add. I've now added all the folders except the ones I really want indexed and I trying it again.

Hopefully, it will work.

I assume there is no log for spotlight where it would complain about a file?

I also ran the following the last time it got stuck:
fs_usage -f filesys <pid-of-mdworker>
No files were being displayed, so it really looks like it hangs when it gets to a corrupt file. I will run this command from the start this time and I will look at the last file it accesses before it hangs. Hopefully that will show me which one it causing the problem.

I figured out which folder it was.

In the terminal, I ran this command while spotlight was re-indexing:

fs_usage -f filesys mdworker

As expected, it was outputting all files it was indexing. When it reached 85% and estimated 8 minutes left, it started printed hundreds of PAGE_IN's. I quickly scrolled up to look at the last file it was working on. It quickly got out of my scrollback buffer (which is set to 2000 lines). Every now and then it would list another file in that folder and non-stop PAGE_IN. I check the estimated time for the indexing and it was going up fast. So I quickly went and moved the suspect folder into a directory that I put in spotlight's privacy. Pretty much immediately, the PAGE_IN went away and it was listing more files. Within a minute or two the estimate was back down to 8 minutes.

It then finished without problem.