I have a similar issue on my neck, my beard grows in all sorts of different directions. In order to get a clean finish requires multiple passes in multiple directions.
Usually straight down, then straight up, then left to right and a right to left. That usually gets everything.
Occasionally a diagonal pass is required as well, but I try to avoid any more than 4 passes as it starts getting uncomfortable.
I have found though that a quite aggressive first pass can help the subsequent passes, so I often dial the razor up for the first pass.