Duelo quest is bugged. Make sure the phantom weapon you find to pull out of the ground matches the weapon mentioned in the current quest objective!
Just found this out after having it glitch one of the fights over and over again. The quest will tell you to pull a phantom weapon from 3 different shipwrecks on the map in a certain order: Longsword, Polearm, and Rapier.
However, sometimes your bird will guide you to a weapon later in the order than what you actually need to pull. In my case, it directed me to the phantom rapier when the objective was for the polearm. It will let you pull out the rapier and start the fight, but when it begins you won't be able to pull out any weapon or fight back at all.
Hope this helps someone!