Spotlight vs. updatedb/locate

Jick · Jun 17, 2011

I've spent an hour searching for any existing thread on this topic, to no avail, so I am taking the desperate measure of opening a new thread; if this offends anyone, I apologize and ask for a suggestion of an alternative.

This may be considered a complaint, but I sincerely wish to have a serious discussion on this topic, so I ask your patience. It will be a bit long-winded, but the alternative would be mere sniping, which serves no purpose.

First a little transparency: I am a long-time Linux user and I don't shy away from the command line interface, so those who prefer to use their mouse for everything may as well stop reading.

Like most folks of my ilk, I have always depended upon the UNIX locate command to find the location of files, e.g. via #locate myfilenamefragment where the argument is hopefully self-explanatory. The locate command relies upon a database of all the filenames on the system and their locations in the directory tree, which in turn is generated by the updatedb command (as superuser). Both commands are available in OS/X, but there have recently been some changes:

Namely, updatedb can no longer access the /Users directory tree, which is of course the most important one. After reading various tangential discussions on this topic I get the impression that this change is attributed to concern for security (one User should not be allowed to see what files exist in another User's directory tree); you will forgive me for being skeptical of this explanation, since (a) most Macs have only one User; and (b) concern for absolute user privacy has not marked most of Apple's policies to date.

The same readings made it clear that any reasonable and educated user would naturally prefer the infinitely superior Spotlight over the crude, ancient, brain-dead locate. Well, OK; so I set out to educate myself. These things I have learned:

1. Spotlight uses up a significant fraction of my CPU time searching through every file on my system and building not just a database of filenames and locations, but a database of the contents of the files. (Privacy? What privacy?)

2. Spotlight's database uses up 5 GB of disk space. 5 GB! I can think of other uses for those GB.

3. Spotlight will not find "hidden" files (those starting with a period) at all, and apparently will not look in system folders like /private and /usr without some serious hacking in well-hidden configuration files. (I found a description of how to defeat this exclusion on another website, but it was for 10.4 and the suggested configuration files do not seem to exist under 10.6 -- evidence that such interference with the "proper use" of OS/X is no longer tolerated).

So where does that leave me? I can always just retreat into the Linux distro running in VMware on my Mac and forget about OS/X, but I have paid good money for my Mac-OS/X software and I'd like to be able to use it. The issue boils down to this: Whose computer is it, anyway? I once considered buying a NeXT back when it was the first UNIX-based Jobs project, but I was put off by all the rules forcing me to build directory trees the way Steve liked them and give things names he would recognize. It seems we are back on the same track with OS/X.

If I am truly alone in my reluctance to be told how to use my computer, perhaps my love affair with the Mac is doomed to divorce. If there are others like me who prefer to do it our own way regardless of how uncool it is, please let me know I'm not alone.

cwa107 · Jun 17, 2011

Not sure what this has to do with "Community Suggestions and Feedback". Please be aware that we have nothing to do with Apple, Inc.

Moved to more appropriate forum.

Jick · Jun 17, 2011

cwa107 said:
Not sure what this has to do with "Community Suggestions and Feedback". Please be aware that we have nothing to do with Apple, Inc.

Moved to more appropriate forum.

Sorry, I thought it was a perfect example of Community Feedback; I guess the forum title means "Feedback on this website" rather than "Feedback on Macs". My mistake. I know you are not representing Apple, Inc., but I thought you were providing a forum for Mac (i.e. Apple) users.

I just want to find out what others think about the issues I raised.

cwa107 · Jun 17, 2011

I think when you switch platforms, you tend to try to force the destination platform into being the one that you'd left. Old habits die hard after all. I also think a lot of people tend to look at another platform, see a difference and say "that's wrong". Just because it's different, doesn't make it wrong. We usually switch platforms because we want a change - so don't be surprised when it is indeed different.

With that said, I have always found Spotlight to be fairly weak in terms of its search capability and was shocked to find that there was no option to force it to dredge through an entire folder structure with options similar to what I was accustomed to in Windows XP. But over time, I realized that whenever I hit Command+Space, either to use it as an Application launcher, a calculator, a dictionary, or just to look for a file name or keyword, I nearly always found what I wanted.

For everything else, I use a nice little free app called "EasyFind" that will indeed dredge every last nook of the file system.

Now, why Apple has limited locate's ability to search the user container at all is definitely confusing to me. It's not a tool I've ever used personally, but I'll take your word that it won't search the users directory. I would tend to think that in cases where you wanted a comprehensive search, you'd need to run it with escalated privileges to begin with (i.e. under sudo), so why then it would be limited is beyond me - even with the "security" explanation.

Regardless, I can think of more significant nits to pick if I had audience with Apple (like why OS X still doesn't have the ability to dynamically change printer ports). But search isn't one of them. If it were, I think I'd just disable Spotlight and give Google Desktop a spin.

vansmith · Jun 17, 2011

If one piece of software is going to bother you this much, perhaps you would be better served going back to Linux. You went from an OS that is more customizable than most desktop operating systems to one that is, in many respects, hands off (unless you're willing to get your hands dirty which you are).

As for your concerns, you seem to be misinformed. Looking at /usr/libexec/locate.updatedb, you'll see the following bit of code:

Code:

: ${mklocatedb:=locate.mklocatedb}       # make locate database program
: ${FCODES:=/var/db/locate.database}     # the database
: ${SEARCHPATHS:="/"}           # directories to be put in the database
: ${PRUNEPATHS:="/private/tmp /private/var/folders /private/var/tmp */Backups.backupdb"} # unwanted directories
: ${FILESYSTEMS:="hfs ufs"}                      # allowed filesystems 
: ${find:=find}

case X"$SEARCHPATHS" in 
        X) echo "$0: empty variable SEARCHPATHS"; exit 1;; esac
case X"$FILESYSTEMS" in 
        X) echo "$0: empty variable FILESYSTEMS"; exit 1;; esac

# Make a list a paths to exclude in the locate run
excludes="! (" or=""
for fstype in $FILESYSTEMS
do
       excludes="$excludes $or -fstype $fstype"
       or="-or"
done
excludes="$excludes ) -prune"

case X"$PRUNEPATHS" in
        X) ;;
        *) for path in $PRUNEPATHS
           do 
                excludes="$excludes -or -path $path -prune"
           done;;
esac

Note that the directories included are everything under root. Also note that the folders specifically "pruned" are mostly temp folders. Note that there are no references to the /Users folder. In fact, I've attached a screenshot demonstrating that locate found files in my home directory:

Given that the updatedb script is just a script, you're also welcome to modify it.

As for Spotlight, if you don't like it, disable it:

Code:

sudo mdutil -a -i off

There's no need to be mad at Apple here for problems that aren't really problems.

Jick · Jun 17, 2011

cwa107 said:
...
For everything else, I use a nice little free app called "EasyFind" that will indeed dredge every last nook of the file system....

That does the trick, although I'd love to be able to just type "EasyFind <stringfragment>" in the command line (where I spend a lot of time) instead of having to do everything through a GUI. Thanks for the tip! At least I found my files.

Jick · Jun 17, 2011

vansmith said:
If one piece of software is going to bother you this much, perhaps you would be better served going back to Linux. You went from an OS that is more customizable than most desktop operating systems to one that is, in many respects, hands off (unless you're willing to get your hands dirty which you are).

I recently complained about the radical changes to Ubuntu's default desktop look and feel, to which they replied, "If you don't like it, install another distro!" I guess I am just a complainer, and no one else bothers, so I should shut up.

vansmith said:
As for your concerns, you seem to be misinformed. Looking at /usr/libexec/locate.updatedb, you'll see the following bit of code: ... Note that the directories included are everything under root. Also note that the folders specifically "pruned" are mostly temp folders. Note that there are no references to the /Users folder. ...

Given that the updatedb script is just a script, you're also welcome to modify it. ...

Encouraged by your assurances, I rooted around a bit and found that while I can no longer just say "updatedb" as root (I have a habit of leaving one terminal window in su mode and using it for all root actions, rather than prepending "sudo " all the time), I can get it to work from my user account by saying "sudo /usr/libexec/locate.updatedb". So you are right and I have egg on my face.

I've been trying to relocate the posting that suggested it was a privacy issue with /Users, to no avail; it hardly matters now that I've got my beloved locate back (Thank you!) but here are the error messages I got when I just said "updatedb" as root (which worked fine until 2011):

Code:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: Permission denied
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Permission denied
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Permission denied
find: .: Permission denied

From there I went (via Google) down a trail that led to the /Users claim.

Anyway, as Roseanna Rosannadana would say, "...Never mind."

vansmith · Jun 17, 2011

I will admit that the first time I tried to update the locate db (Linux user here as well), I was lost. Finding locate was easy but it took a while to figure out that not only was the db update tool not in my path but actually wasn't called updatedb.

If you update your db on a regular basis, you might want to look into whipping up a launchd process. This looks to be a good place to start if that's something you're interested in.

cwa107 · Jun 17, 2011

Well, this turned out to be a great thread! Always nice to air out some good solutions to even the more obscure problems out there.

Jick · Jun 17, 2011

Ooooh, nice! (I won't start complaining that they didn't just call it cron. ;-)
Thanks again for your help!

vansmith · Jun 17, 2011

Jick said:
Ooooh, nice! (I won't start complaining that they didn't just call it cron. ;-)
Thanks again for your help!

Haha, Apple did used to use cron but switched to using a homegrown solution (launchd) with OS X 10.4. That said, cron still seems to be hanging around, perhaps for legacy reasons (I don't actually know why it's still there). I suppose you could add a line to your crontab if you chose to go that route. Otherwise, you could always symlink /usr/libexec/locate.updatedb to something in your PATH or just add /usr/libexec itself to your PATH.

Spotlight vs. updatedb/locate

Jick

cwa107

Jick

cwa107

vansmith

Senior Member

Jick

Jick

vansmith

Senior Member

cwa107

Jick

vansmith

Senior Member

Shop Amazon