I'm singlehanded 99.9% of the time. Only ever sailed The Red Witch once with crew who could be considered "experienced". Of all the improvements I've made to TRW (pretty much all have been mentioned here), I really can't think of any I did that necessarily made her more "single-handed" friendly, maybe with the exception of the tiller tamer?
(Caveat: I'm 6' 2", 165, and can pretty much reach everything in the boat no matter where I'm sitting. Mast raising is pretty easy as well)
I run the usual tiller tamer from D&R (Davis brand), and I rarely even tighten the knob. One end of the line is fixed and the other end goes through a clamcleat w/ fairlead. Keep that line snug with tiller amidships, and it will stay there with the knob loose, but it's very easy to just push the tiller to one side or the other - it gets loose as you do so. For heaving to, I just push it over to port and reset the line in the cleat.
As far as setup time goes, I guess everyone has their own "system", and having help doesn't help, even if they're experienced. I don't have a "checklist" per se, but I try to do it the same way every time (setup and takedown), and the best I can do is about 15 minutes from arrival at the ramps to leaving the dock. But you're right GL: retrieval always takes longer. Is it a case of separation anxiety - not really wanting to end our water time?
Now, as our 2-year-old grows up, if she displays an interest in learning how to sail, I'm gonna have to go from single-hander to instructor, so all of the above will most likely go out the window, and all bets will be off!