Anil John
Making Digital Services Secure and Trustworthy

Anil John

Avoiding the Echo Chamber when Building a Research Agenda

 Tweet  Share  Comment  Print  Email

In deciding what research areas to focus on and invest in, it is critical to combat confirmation bias which leads us to collect skewed, self-serving information. Consulting domain experts is often the most employed practice in combating this bias. That, however, comes with its own challenges and is an incomplete approach at best.

Avoiding the Echo Chamber when Building a Research Agenda

Confirmation bias is fed by our ease and comfort in interacting with communities and people who share our world view rather than those who do not.

So a technical organization seeking to put together a research agenda for a particular domain is naturally going to gravitate towards the experts in that domain to identify the capability gaps.

But if you ask an expert in authentication technologies about capability gaps in the domain of identity management, the answers will invariably center around strong authentication technologies with a bit of lip-service thrown in for adoption rates, user experience and privacy.

As such, the view points of experts are going to be shaped by their area of expertise and it will be hard to get them out of their narrow frames if you are seeking a more broad based perspective.

Casting a wide net to understand problems

I personally believe that in order to build an effective research agenda, you need to step back from asking experts about capability gaps in a particular technology area and start asking normal human beings about the problems and pain points they are encountering when using the technology.

One of the benefits of studying technology and being on the cutting edge is that you can envision what it might be. Now it's important to ask people what their problems are, not what products they want.

Bob Maresca, CEO, Bose Corporation

Just don’t expect it to be an easy conversation.

As an example, remember the last time you ended up being “family technical support”?

Now, imagine having a conversation with that family member about your area of deep technical expertise that touches their life, and trying to gain an understanding of how you can make the impact of that technology in their life be easier/better/more secure/more privacy respecting etc.

That should give you a sense of how completely sideways this conversation can go as you are seeking to understand the problems!

Leveraging deep expertise to solve problems

But once you gain that broader perspective and understanding of the problems, it is time to utilize the strengths that experts bring to the table.

The simplest and most intuitive advice we can offer [...] is that when you’re trying to gather good information and reality-test your ideas, go talk to an expert. Here’s what is less intuitive: Be careful what you ask them. Experts are pretty bad at predictions. But they are great at assessing base rates.

Decisive: How to Make Better Choices in Life and Work

While experts are typically bad at predicting the future, they are the perfect set of people to turn loose on the problems identified in the broader conversation.

Given their excellent understanding of the art of the possible in their domain, they can help you in framing out possibilities and approaches for solving problems that are relevant to their domain.

Does this sound like something that takes an investment of time, toil and treasure? It is, and I do not want to downplay that. But I do think that the long-term rewards of solving problems that truly matter to people are well worth it.

For those interested in learning more about how elements of this approach were used in a practical, public sector setting, I would urge you to read up on how the Canadian province of British Columbia conducted its Digital Services Consultation when deploying its “Services Card”.

In short, do not ask people what products or services they need, ask them for what their problems are. Then provide that information to experts in the domain who can help identify the technology and innovation needed to solve them.


Did you find this interesting? Don't miss any new posts. Sign up to automatically receive them now!

I will never share, rent, or sell your information to anyone. Cancel anytime.

This blog post first appeared on Anil John | Blog ( The opinions expressed here are my own and do not represent my employer’s view in any way.

By on |

Continue The Conversation ...

I would love to know your thoughts on this blog post.
Meet me over on Twitter or LinkedIn to join the conversation!

I am a Public Interest Technologist. I help technical leaders make digital services secure and trustworthy. Learn more »

Free Updates

I will never share, rent, or sell your information to anyone