You mentioned it helps if it's primed... do you just turn the key to 'ON', wait for a second or two, and then try starting it?
Also, is it harder to start when it's been sitting for a few hours versus if you just popped into a store for 15 minutes and came back out to start it?
If so, it may be the fuel pressure regulator. I have the same issue, but if i prime it by turning the key to ON for a couple seconds everything is fine, so I haven't investigated further. You could also check the fuel pump relay like
Towncivilian suggests in this thread.