I'm saying that if the rudder swings back, you'll definitely feel additional force in otherwise same conditions.
Not saying that a change in conditions couldn't change the rudder forces as well.
If you sail with a non-swinging rudder, the force you need to apply at the tiller should relate to the force that the rudder needs to supply to counteract the forces caused by the sail and the hull.
If you are sailing the boat flat (not or not excessively heeled) there shouldn't be a net steering contribution from the hull, but if you are heeled very far, the immersed hull becomes rather asymmetric and will definitely steer your boat in a curve all by itself (can be used in light winds to help the boat tack).
Sheeting in the main or freeing the jib results in the boat wanting to turn upwind (causes "weather helm"). When the wind gets stronger, the apparent wind moves back. (Contribution from boatspeed becomes less). If sail trim stays the same, that could increase the tendency to round up (more weather helm). However, once the boat speeds up, the apparent wind would move forward.
Hard to separate the contributions from the sail; if you want to be sure, you'd better test this a bit. With a motor at your disposal, you can push your boat to and past hull speed while it's level (and no contribution from sails). If the uphaul gets slack by a bit that would tell you that the rudder moved.
I would definitely fix the hydrodynamics of the forward attachment; there's no question that you are paying a constant penalty there.