~/Library/Keychains is…a file?

No, it’s actually not a file, it’s a directory. At least it’s supposed to be. But, multiple reports from multiple lab maintainers had been coming in that included error messages from applications which were trying and failing to access the user’s keychain store. I started troubleshooting by repairing the keychain with /Applications/Utilities/Keychain Access.app, but that did nothing aside from provide more of the same error messages. I suspected keychain corruption or possibly mucked up permissions, so I opened Terminal to take a look. ~/Library/Keychains was a file!

It took me a while to figure this one out, and now that I know what it was, I’m admittedly a little embarrassed. This is one for #macadminshame. For the machines I maintain directly, I manage the user environment with Local MCX. That’s not a technology that my lab maintainers are comfortable with; they’d much rather login using a special account, make their changes, and then rest comfortably knowing that those changes would be present for everyone who logged into their lab machines. I get that – we all have better things to do than learn about complex technologies that we’ll use less than 3 times annually. So I wrote a logout hook for them that ran when this special account logged out. The script empties the trash, clears caches and logs; generally cleans up the home directory for that account. Once cleaned, it bundles that home directory up in an installer package which I then import into Munki and deploy to the rest of their lab machines for them.

Of all the cleanup tasks that I had been doing, one very important one had slipped right by; ~/Library/Containers/. If you happen to update your Non_localized.lproj, English.lproj, .lproj directory like this (which isn’t recommended), please be sure to purge the contents of ~/Library/Containers.

I didn’t bother trying to figure out which container was touching ~/Library/Keychains because once I realized my error, I knew everything in there needed to go anyway. Moral of the story: Profiles/MCX is the way to go, but if you can’t, make sure you’re not putting anything in /System/Library/User\ Template/.lproj/Library/Containers/.

Also, if you find yourself already in this scenario (hopefully I’m the only one who will), you can fix existing home directories by deleting ~/Library/Keychains (as long as it’s a file, not a directory!) before the user’s next login.