Hmm, the application might not start as I think some apps check to see if their data is intact before starting - they would notice the extra files and wouldn't open.
EDIT:
I just tried this with an App I figured I could do without (Chess) and put some random files in the first level of the package (next to the Contents folder) and Chess started without a hitch, so yes - you can hide your porn there. :headphone
Shop for your Apple, Mac, iPhone and other computer products on Amazon.
We are a participant in the Amazon Services LLC Associates Program, an affiliate program designed to provide a means for us to earn fees by linking to Amazon and affiliated sites.