I've done a lot of solo sailing in a lot of conditions. I'm big - 235 lbs., 6' - but it's all moveable ballast :lol: and thus controllable, and I've really enjoyed the extra amount of control I have by just, for example, shifting my butt in or out from the centerline a few inches, or scooting fore and aft a bit. It's especially noticeable when I'm hiking.
I remember being very surprised when first having guests in the boat - my family. The boat felt much more stable and was much less twitchy, less responsive to people moving around. I found this to be true with four adults as well, which was more weight. This extra people weight was back on the seating, not down by the mast step, and I would think the location was as important as the extra poundage. The boat felt bigger and more stable.
So, yeah, try putting some extra weight in the boat. Think of having another person sitting with you and where they might be. What occurs to me is maybe those tube bags of sand you get for cars in the winter (now would be a great time to buy!) and putting them amidships beside or behind the centerboard trunk. You could, with your foot, nudge the bag slightly to help give you the trim you want, kind of like the old sandbagger sailboats of past centuries:
http://en.wikipedia.org/wiki/Sandbagger_sloop
DO keep in mind that extra weight will be putting the boat lower in the water and thus possibly taking on more water over the gunwales in gusts.
Carlos