Pay attention to how detailed their response is and their aptitude for creating robust specs.
While most of the leadership roles (CXO/VP) in any organisation comes with some kind of explicit or implicit authority or veto powers, Product Manger (PM) role are generally not bestowed with any of such rights. PM have to earn team's trust and respect over a period of time. It create all kind of leadership challenges for PM from perception of his role in team & organisation to his ability to say 'No'.
This to me is the heart of product management. You may have said a lot of great things, but if you don't have a great answer to this question, we are done. There are lots of good answers. But I'm looking for a mix of things. First and foremost, have you put thought into this. If you just ramble off an unrelated list of things, we are done. Second, I want to understand how you balance all the competing inputs: user feedback, business needs, the CEO's pet feature, engineering favorites, time to build, cost, market needs, and so on.
This is a great question for gauging how well someone understands the business context in which they work and the market in which their product sits. It's also a good measure of critical thinking. If you want to move on to an in-person interview, you better nail this one.
One of the most important things that a product manager does is listen to needs of stakeholders and understand the motivation behind those needs. If that person doesn't have a well thought-out process for collecting information to make good decisions they will never be successful in the role. Having a pre-determined information gathering process also demonstrates their organization and process skills, which are paramount.
The candidate's attitude can be a key consideration. Obviously a mobile PM has to be at least somewhat a type A personality, in order to get results. However, if the candidate disrespects his past co-workers, monopolizes conversation within the interview, interrupts constantly, or more, these can definitely be red flags in the hiring process.
This is a common variation of the strategy question. Case study questions can be tricky. To prepare, research the market that your target company is in, as well as adjacent markets and competitors. Review recent tech business stories and analysis to understand current business strategies in play and what issues tech leaders are grappling with. If you are caught off-guard, you will at least have some analogs you can draw upon.
Like the product design questions, start your answer with stating your assumptions about the current business strategy and goals of the company in question. Common case study questions involve new market entry-Michael Porter's Five Forces framework can be a good starting point.
Discuss how the new market or segment fits with the company's core business and complementary business opportunities. Understanding the company's underlying value proposition can help put a seemingly unrelated business opportunity into context.
Many product managers can be too prescriptive in 'how' things are done while not allowing for flexibility as development highlights difficult areas of execution along the way. Good product managers know how to make concessions in certain areas which allows for the balance of development time and covering core needs of the user. Specifications should be written with this in mind.
Easiest way to earn your team's trust and support as PM is to develop intimate customer & market understanding. Having a well developed spidery sense is not only a huge plus but an requirement. On various occasions you have prove that you know and care about customer more then anyone in the organisation. This can be tricky thing to accomplish as many PM rely on others for market & customer information. In other cases PM job description may fail to communicate your responsibility as customer evangelist clearly. Keeping all this in mind ability to filter noise from signal and developing strong intuition about market becomes the most critical skill for PM's survival.
Now coming back to interviewing questions I think any question that help you gauge a candidate on above skill parameters is a good question.
Moreover I believe organisations as well as products go through an evolutionary life cycle and each stage requires different attitude and temperament from new PM. It would be better if you keep your organisation and product goals transparent in the interviewing process. Let the candidate evaluate product path and if it'd not fit into your organisation goals communicate your desired product trajectory to him so that he can answer your question according to new expectations.
Product design questions test the applicant's ability to think on their feet and create a full product or feature. New product design questions can be very high level ("design a lamp"). Start with identifying the goal of the product-if the interviewer won't tell you, state your assumptions so that you have something to build on. Decide which metric or business driver you will impact most. From there, identify the possible users for the new product or feature. Select the one that seems most relevant.
You won't have time to cover everything in the answer. Once the user is selected, move on to use cases, goals, and scenarios for that user. The use cases should naturally result in a set of features for your new product. Prioritize these and close by linking back to the goal of the product, the business strategy, and the user needs you are meeting. Show you aren't afraid to color outside the box by including a range of features in the product or tackle a novel problem. Present a range of ideas ranging from mundane to outlandish and demonstrate that you can generate a broad range of ideas and decide among them quickly.
If they have a process for pulling customer insights, and mention market research tools and how they have used them in the past, they most likely understand generating customer feedback is important to inspire good products.