A knife and/or line cutter should be part of your standard gear for various reasons as mentioned.
IMO, any time you're in an environment with big predators there should be some consideration, but specifically bringing a knife to fend off a shark is the equivalent to the old saying of bringing a knife to a gun fight.
Here's my advice... You need to identify what any predator's behavior is... If you see a shark, don't panic, watch it. Keep your eyes on it. If it knows you see it, it knows the gig is up for any ambush tactics, which is how they hunt. Probably 99% of the time they are just passing by and will continue on.
If the shark hangs around, starts swimming a little erratic, darting back and forth, dropping its pectoral fins, arching its back, it's time to get out of the water. That is a display of aggression. Some sharks, like bulls are full of piss and testosterone and (personal anecdote) can be territorial. If there are more than one shark, this could trigger a sense there's a feeding opportunity. Some species do hunt in packs. I've seen this behavior in late summer with bull sharks, again personal anecdote. If I see two or more sharks of the same species, I'm thinking in the back of my head about making my ascent. If they start circling me and are showing no signs of moving on, I'm calmly going up. Being with a buddy will be helpful to watch each others back in the case of multiple sharks. The bottom line is you need to be able to identify when it's time to get out of the water.
Someone earlier mentioned a selfie stick. In terms of actual defense, in the absence of bringing a gun to a gun fight, that's your best option. Even if I had my speargun with me, my first reaction would not be to shoot a shark checking me out, it would be for me to spread out a little, make myself big and intimidating, slowly swim toward it to show dominance and use the selfie stick, speargun or whatever long item as a tool to establish distance from the shark. Sharks are not stupid animals, they don't have a "fight to the death" mentality like some animals do. They will practice self preservation.
In the extremely unlikely event I was attacked by a shark and all I had was my 3" knife, you can bet I would be trying to slam it into its eyes or gills. The nose is too close to the business end and it's how many shark attack survivors end up asking their wives to open their ketchup bottle the rest of their life.
The bottom line, if you see a shark, you watch it. If you don't feel comfortable with the situation, get out of the water. If you hunt, you will encounter sharks looking for a feeding opportunity. If you're just diving, keep your camera ready.
Here's an incident where a diver used his knife to fend off two sharks. Keep in mind he was spearfishing and then started panicking, both will trigger a feeding response.