Yeah, just a few pounds can trim a hull.
In the end, when rowing, you want the boat to glide straight through the water, and NOT have the boat 'weathervane' in the wind.
this weathervane effect happens when the hydrodynamic center of force is significantly fore/aft of the center of boat. To keep it pointing forward, you want the pin of the weathervane to be foward, so that the aft of the boat follows the wind.
Moving the CB up or down a an inch or two will slide that COF fore or aft.
Since I was sitting with my butt foward of the centerboard jib cleats, with my back to the bow, I could easily trim the CB when I noticed Weathervane drift.
You are right though: if you have more weight on either side of the boat, you won't go straight... I'm pretty sure that this because the curve of the hull will create a lift on the high side, since there is typically more curve under water on the low side... if that makes sense.
So: sit in the center, weight a little forward of Hydro COF, use the CB to trim as needed, keep the oars applying force behind the hydro COF.
stow the tiller, all it does is make the turns more dramatic, and harder to control, honestly.