Before a few years ago, this work was done by market researchers, a highly professionalised industry. Practitioners typically have a qualification from CIM, MRS and/or a psychology degree. Most research team leads have PhDs in a related field. Training takes many years.
As a researcher, I trained under two research leads at a top university. We didn't throw a report over the fence but developed the interview guide with product teams, deftly teasing out requirements while stripping out bias from the questions.
The product teams would sit behind the glass, Mad Men style, during interviews, and we would pause the interview near the end and pop our head around the door and ask if the team had any additional questions.
We would then go through the results in a debrief workshop with the product team and turn the insights into product requirements.
Now, almost all of this expertise, experience and basic knowledge is gone.
The tech industry has a spirit of anti-intellectualism that holds the skills gained from an eight-hour online course equal with that gained from years of training by experts.
The result is disastrous. I recently ran a reverse discovery project with a team on a mature product and found that they were wrong about just about everything. This is far from the first time I have seen this.
Unless you have a clear understanding of the needs and motivations of your customers, you cannot answer the most basic questions of strategy.
We are starting to witness the failure, at scale, of entire industries built around a basic misunderstanding of how to run a business.
I have noticed that companies are starting to hire Chief Revenue Officers (where the hell is all the money going?) and now security architects (you can't keep shoving out any old crap to make your burndown charts look prettier), and finally, eventually, we will see the return of properly trained consumer researchers. It's not whether Scrum has failed or SAFe has failed or even waterfall has failed - it's ALL failed.
The appallingly amateur (in the most literal sense) approach to user research is just reflective of a general culture of irresponsible recklessness.
