What is a way to communicate these preferences without being rude?
Make it clear that they're your preferences, without implying that anyone who doesn't share the same preferences is wrong or foolish.
For example, in this question, you originally wrote, "It requires a large amount of research to answer correctly. If you answer with the first thing that pops into your head, you're probably going to get it wrong." A better way of phrasing this would have been, "I'm only interested in research-based answers; please don't post an answer unless you know a lot about the subject."
To see what I think are polite ways of expressing such preferences, see most of these questions. For example, in this question, I wrote, "I'm looking for answers based on connecting clues from the text to actual facts about politics. Authorial-intent answers with quotes from Lukyanenko would also be welcome, but please don't post answers just based on your own political views unless you can back them up with something more concrete. This is a literary question, not a political soapbox." The last sentence of this is probably the least polite I've been in a question here, but (IMO, at least) it still stays within the realm of "firm without being rude".
Is it acceptable for people to write answers that don't conform to those guidelines, even if their answer is arguably a better way of approaching the question?
It depends. I would say there's a useful distinction to be drawn between answers which simply miss the point and fail to answer the question as asked (these could be downvoted and maybe even flagged as NaA) and those which make a valid "framing challenge" to the question by presenting a good argument for why X instead of Y is the main issue at hand and answering that instead.
We can probably learn a few things here from our comrades at RPG.SE, who wrote the following guidance for how to write a good "frame challenge" answer:
Provide your critique as part of an otherwise legitimate answer. In general your challenge should not consume more of your answer than the actual answer does.
Do it carefully and support your challenge - "I prefer different things" is not really a good reason, remember this isn't a forum and the goal of the site is to solve people's problems, not to wave our own personal freak flag around
Don't get upset if you get voted down because the community thinks you overstepped your bounds. You know you're taking a risk by doing it instead of just answering their question.
See also What is the XY problem? at main meta for more discussion of related issues.
What is the best way to resolve debates between the asker and the answerer about what types of analysis are appropriate?
Ultimately, what makes the most appropriate answer to the question is the asker's decision, but on the other hand, a good answer can change the asker's mind.
Let's say I post a question asking specifically about authorial intent, and you answer it saying that authorial intent is the wrong thing to be considering and giving an analysis based on something different. Then either you'll convince me that your answer is actually the right way to look at things, or I'll simply not accept (and maybe downvote) your answer.
As always, the asker has the prerogative to choose which answer resolves their question the best. That means the onus is on the answerer, if they provide an answer outside the stated preferences of the question, to convince the asker that their answer is the 'right' one. If they fail to do so, the asker can simply not accept their answer (though of course they can't prevent others from upvoting it).
You ask about resolving debates, but unless there's some bigger policy at stake or some other reason why third parties should be interested in the outcome (in which case, take it to meta), I think this is something for the asker and answerer to hash out between themselves, in chat if necessary.
