I might have to try reading more 2nd person stories. I generally stop reading them immediately as I find them annoying (and usually badly written) but perhaps I'll give them a chance again and see if I change my mind.
The writing is key. Your feelings aren't entirely wrong. A poorly told story completely ruins the feel no matter what, and it's especially true for 2nd person. The greater immersion makes the flops harder on the mind. But like I said, if it's well done, it's magical. Some people object to the ambiguous physical descriptions used in these to let the author truly become the protagonist (myself included), but even so, a solid prose makes it manageable.
But I don't mean only stellar syntax and dialogue. It has to be logical. Again, you're not entirely wrong. If the writer oversteps a logical extension of a given choice, it dulls the immersion. The problem there is writers can forget this and dictate the character's actions by themselves. The standard in fiction is clearly defined characters who can be predicted by anyone who knows their characters. A story with established characters shouldn't offer any of the choices a choose your own adventure would, but if the characters have not yet been established and the choices dictate their personalities we should have breaks to make those decisions. A 2nd person story's protagonist is always the latter, and some writers forget this. That said, once the choice is made a logical extension would be analyzing why the character is reacting this way and can be expressed without any other choices preceding it.
On top of that, some writers do play god in a more suitable capacity. If the reader decides to forgo a chance, some writers can decide they don't get it back and create a new situation, some requiring no choice in their own right. A simple example, the end of a doctor's exam. Once it's been established the protagonist won't be stopping the doctor, some writers will mush the character out of the exam room, others will have the staff stop the character. This too is acceptable, since it's not the character making this decision, it's the situation and common sense; since the character already decided to end this exam, the character will naturally clear out unless someone stops it. In fact, the exact same thing can be said about students at the end of a class who decided to not talk to the teacher for extra help and office workers who decided to not stay late.
It all comes down to how far is too far, balancing the logical behind the choice and situation against separate choices. A 2nd person story needs to stop when the logic behind the previous choice and the current situation can't carry the story any further on its own. This way, it expresses what's already been established without establishing anything on its own. It's a little tricky, and some writers forget this, but a proper story is magical.
That said, some stories have the wrong format. Some stories attempt to have clearly defined characters, yet they present them in 2nd person and generate contradictions - sluts get a prude choice, exhibitionists question situations for more than legal ramifications, etc. I'm not against defining the physical appearance - it makes for consistent storytelling and erotica itself is about exploring fantasies - but writers that dictate protagonists' personalities defeat the whole purpose of the perspective. Add to it, we have writers who give the reader the ability to choose how other characters react to situations, which makes no sense if readers are only controlling one character; break this rule to establish the character's gender and situation, but after that it has to be the reader who makes decisions.
I'm not going to say this perspective has the best writers - in fact I think it's the hardest to write for - but I love the concept and it epitomizes interactive fiction.