Ah, I see the confusion there.
The double is trying to disarm your holy symbol. So because it isn't a weapon, and easy to grab, you get a -4 to keep hold of it when using the opposed rolls in the disarm. However, your double was attacking unarmed, so he got a -4 too. Hence you guys were just rolling completely normal opposed attack rolls effectively.
However, before all that, he provokes an attack of opportunity coming that close to grab your symbol. So you can still use a sword to deliver a completely serparate attack. If it succeeds (which it would if it was a lethal attack) and deals damage (lethal or non is fine by me, but attempting non-lethal will miss in this case) then you stop the disarm attempt before it gets to oppsoing rolls.
http://www.d20srd.org/srd/combat/specia ... htm#disarm
It is a little confusing, sorry! Basically, if anyone (you included) tries anything cool, more often your enemy gets a completely separate attack of opportunity to stop the attempt dead.
Good news, in the case of your grapple attempt, Haynuus is totally unarmed AND grappling, both of which mean he does not threaten you, so can't make AoOs.