The best fish market is in Gustavia. Again, head back towards the airport. At the roundabout above the airport take the first right, and head down the hill into Public. Continue through Public and the commercial port until you enter Gustavia. The very first building on the right as you enter is the fish market. The fisherman are there early, and gone as soon as the fish is gone. Bring cash, and be prepared to speak French or speak with your hands.
Your best choices are likely the big fish. Daurade is Mahi Mahi, Thon is Tuna.