A Commemorative Event

[muchtooarrogant]

LJI Week 1: Quality
June 15 marked the ten year anniversary of working for my current employer. This was both gratifying and slightly amusing to me because of a comment made by one of my former coworkers when I left my previous job.

"An accessibility job for a large mainstream company?" they reportedly scoffed. "That'll probably only last a year, if that long."

Hello, still here!

Oh, don't get me wrong, it's definitely been a bumpy ride at times. Our team has always been that troublesome puzzle piece that company leadership wasn't quite sure where or how to slot into their operations. It was a bit like being part of a jigsaw puzzle where the assembler could change a piece's location, but also its size, and shape, and name, and overall purpose. And, don't let the default past tense of storytelling fool you, they're still at it!

The disadvantages to being that puzzle piece are obvious, but believe it or not, there were advantages too. For example, it meant I did the routine stuff like reviewing content to identify accessibility issues, but also worked with developers to design feature enhancements so that the content was easier to understand and navigate. Myself and other colleagues were encouraged to share our knowledge by responding to calls for papers at national conferences, and if we were accepted, got to travel to the conference and present are findings. Without a doubt though, customer research was one of the most interesting activities I got to participate in.

Of course, ten years ago, I didn't know anything about research or usability studies seeking customer feedback. When I was asked to help out with our first study, I sorta figured, you put together a prototype, showed it to the customer, let them play with it for a bit, and then asked a few questions. Easy-peasy! Yeah, not so much.

• What are the research questions your study is trying to answer?


• What prototypes or study materials will be shown to each participant?


• Is there enough time to prepare the prototypes/study materials before the study is scheduled to begin?


• Will participants need to sign nondisclosure agreements in order to view the prototypes/study materials, and has legal approved the language for those agreements?


• Do you plan to compensate participants? If so, how much will each participant receive? Have you secured funding for the compensations, and if so, for how many participants?


• Where will the study sessions take place, and is there a fee to use that space?


• How much time will be allotted for each session?


• How do you plan to recruit participants for the study?


• Will any study participants be less than 18 years of age? If so, have you written a parent/guardian consent form? Has legal approved the consent form language?

You know that old Star Trek joke about the different management styles of captain Kirk verses Captain Picard? With Kirk, the strategy with a planet was beam down and poke any aliens with a stick, and if it was another ship, "Fire phasers!" Picard's strategy, no matter what was in the offing? "Lets hold a meeting."

Yup, you guessed it, I was on Picard's ship. So many meetings!

If you're reading the list of logistics above and thinking to yourself, "Dude, where's the fun?" I don't blame you. The answer was, in the beginning, I didn't have to worry about most of that stuff. I got to bounce potential research questions around with other team members, and then, once we had nailed down what we were trying to discover, I helped write the research protocol. The protocol was important because when you're conducting a usability study seeking feedback, the golden rule you're supposed to follow is to not bias the participants. The prototypes or study materials should be evaluated the same way by every participant, and one of the ways to ensure that consistency was to write a script that would be read in every session.

Now, what about the prototypes or study materials? Should you show those to participants in the same order every time as well? If you're like me and said, "Yes," I regret to inform you that you're wrong.

Think about it this way. You're participating in a research study lasting an hour and a half and looking at three different prototypes for the researchers. By the time you reach the third prototype, the likelihood is that you're going to be getting tired and imagining how nice it will be when the session's over and they stop asking you questions. If I always presented the prototypes in the same order, prototype number three would probably get a pretty raw deal. So, as researchers you counterbalance or change the order of items being evaluated.

Yes, it was a lot to keep track of, and it did get just a bit tedious reading the same script to six or eight participants every day during the study, but the payoff was being able to meet and speak with people with no filters in-between you and their opinions. The hard part, if you can believe it in this age of social media where people seem compelled to find fault with every little thing, was convincing participants to answer honestly and in depth. "We want to know what you like and what you don't like about each prototype, and you won't hurt our feelings in any way by telling us what you think about them."

There were typically three or four of us in the room with a study participant, one person reading the script and asking questions, and the others observing reactions and taking notes. We encouraged participants to "talk through" their experience while exploring a prototype, and I imagine they sometimes may have felt as though they were under a microscope. If they had a pensive expression or made an unhappy noise, one of us would invariably say, "Could you please share with us what you're thinking right now?"

It was time consuming, and exhausting, and while a study was going on, we'd barely had time to grab a meal in-between sessions. Is it weird if I say I miss the experience?

The last study I participated in was during the summer of 2022 at a conference in New Orleans, just after the July 4th holiday. We all wore masks during each session because COVID was very much still a thing. While venturing out for a late dinner one evening, a coworker accidently led me into a hole in a New Orleans sidewalk. Before the conference was over, we had to push the abort button and head home because everyone on the research team, except me, caught COVID. And finally, when I did arrive home, I decided to self-quarantine in a hotel for a few days, just in case I might still be a carrier of the dreaded virus.

Yeah, even with all of that, I still miss conducting research studies.

Dan

Comments

[used_songs]

This was really interesting and I loved all of the details you included that made me feel as though I was learning something. Having been in a job where we had a LOT of meetings, I really identified with that part as well.

[muchtooarrogant]

Thank you, I'm glad you enjoyed it.

When I started, I couldn't believe all the meetings people wanted to hold. After ten years though, I've mellowed a bit, sometimes it is useful to get all the stakeholders together and talk through the details, but ONLY if the meeting is being lead by a really good project manager. I still can't stand the ones that wander along with no purpose--just send an e-mail. (grin)

Dan

[krispykritter]

Congratulations on lasting 15 years when your previous employer didn't have faith! This was very interesting and gave lots of details. Smile.

[muchtooarrogant]

Thank you! I've never really understood the people that jump around from one job to another every two or three years, that's usually when you've actually gotten a handle on what you're supposed to be doing. :)

Dan

[bleodswean]

It's always such a joy to read well written, informative Idol pieces! I learned something here! And I can totally relate to how enjoyable a non-filtered opinion would be to experience without having any sort of emotional / mental horse in that race!

[muchtooarrogant]

Thank you very much. This is kind of a specialized topic, so wasn't sure how interesting people would find it.

LOL As I said in the piece, it's important not to bias participants, but everyone goes in with preconceived notions about how people will feel. Occasionally, while someone is talking, you begin to realize that maybe your notion totally missed the mark. Disconcerting, but after all, that's why you're doing the research.

Dan

[fausts_dream]

Yeah I learned quite a bit from this entry pretty cool stuff, good job as usual Dan.

[muchtooarrogant]

Thank you!

Dan

[halfshellvenus]

Those protocol questions seem tedious, but I can see why they'd be important.

If you're like me and said, "Yes," I regret to inform you that you're wrong.
I also assumed the answer should be "Yes," but you explained very well why that is not always the right idea!

I've listened-in to raw customer feedback sessions on potential and existing products by my company, and they were eye-opening. One caused us to cancel a product. But the more surprising reaction was the guy who said he was about to take a one-week training course on how to use our new intuitive, GUI-based management application.

Never underestimate the stupidity of customers! And someone who DIDN'T underestimate it found a way to make money hosting courses on how to do what we thought was obvious!

[muchtooarrogant]

Yeah, I thought at first I'd write more about the experience of conducting the study in New Orleans, sidewalk holes and all, but I guess some part of me wanted to relive/share all those planning meetings. LOL

The studies that make you change your entire approach can be enlightening, but damned scary as well. I think we generally had a pretty good idea of what people would say, but there were certainly a few occasions that made us rethink our approach.

Thank you for reading!

Dan

[erulissedances]

This was interesting. I hadn't considered all of the different ramifications involved in research studies like this.

- Erulisse (one L)

[muchtooarrogant]

Thank you! Yup, planning the study was always time consuming, so much so that we often had to start meeting and hashing out the details at least six months before it was scheduled to take place.

Dan

[hafnia]

Accessibility work is totally vital, and not something I know a lot about. This was interesting and enlightening. Thank you for sharing!

[muchtooarrogant]

Thank you! It's frustrating sometimes because with all the shifts between leaders, we keep having to explain exactly what it is we do, but I do honestly enjoy the work.

Dan

[marjorica]

Thank you for explaining a complicated process so well and so interestingly.

[muchtooarrogant]

Thank you for reading, and I'm glad you found this interesting.

Dan

[looselips67]

Hello, still here! Lol! Well done!

[muchtooarrogant]

Thank you! (grin) It's certainly been an interesting ten years. A little less crazy now, but they've also reduced the scope of our responsibility. (shrug) Whether it'll last another ten years, that's anyone's guess.

Dan

[rayaso]

This was fascinating. This is a field that I have never had any contact with. I can see why you have kept at. You wrote this very well, making it interesting to read.

[muchtooarrogant]

Thank you! Very glad you enjoyed it.

Dan

[legalpad819]

What an interesting part of your job! I think I'd miss it too- for the change of pace and all that.

[muchtooarrogant]

Thank you! Yes, it was definitely a change of pace, and intriguing to see how each new participant would respond.

Dan

[murielle]

Wow! Okay, I'm clueless. Or I was until I read this. I once took part in something like that. I guess I was too weird. They never asked me back for another study. But I imagined it was such a simple procedure from the interviewers perspective. So wrong!

Now that you have taken me behind the wizard's screen, so to speak, I have a much greater appreciation for these studies.

And I remember how much you have enlightened me on so many subjects over Idol seasons. Thank you.

Another skilled, informative, and expertly paced entry, Dan. Bravo! 👏👏👏

[muchtooarrogant]

Thank you! LOL Behind the wizard's screen, I like that.

With your past study, it may not have been anything about your participation that didn't get you invited back. Another pressure researchers are frequently restricted by is the push for new/fresh perspectives. There are exceptions, as with anything, but that's most often the priority.

Thanks for reading.

Dan

[wolfden]

I found this very interesting.

[muchtooarrogant]

Thank you for reading, and I'm glad you found it interesting.

Dan

[roina_arwen]

That’s a lot of info about studies that I never really thought about before. Congrats on lasting ten years!

[muchtooarrogant]

Thank you!

Dan

[xeena]

I love how I always learn from your pieces and at the same time they are always so much fun!

[muchtooarrogant]

Thank you! Very glad you enjoyed it.

Dan

[inkstainedfingertips]

That sounds like A LOT for sure. But it also sounds fascinating. I think that would be a really interesting process to be part of, start to finish. I can see why you miss conducting the studies. It would be super interesting.

Thanks for sharing this story.

[muchtooarrogant]

Thank you! Definitely a lot of work, but a lot of fun too.

Dan

[alycewilson]

I enjoyed getting this up-close look at your workplace.

[muchtooarrogant]

Thank you!

Dan

[gunwithoutmusic]

Honestly, I did not find myself asking, "where's the fun?" That job sounds incredibly interesting - although I'm sure the day-to-day reality of it is not nearly so interesting. It must have been pretty incredible running those tests, though!

[muchtooarrogant]

You're right, it was very cool observing their reactions and having conversations with them about the prototypes. In most cases, I had been living and breathing the features we were showing them for months, and had some very definite ideas about how "I thought" they'd respond. The participants I came to appreciate the most were the ones who responded unexpectedly or showed us something we had overlooked.

Thank you for reading.

Dan

[static_abyss]

I am very interested in research so reading about this experience wqas great. It all sounds like fun to me, even the inevitably frustrating parts of it.

[muchtooarrogant]

That's what I realized while writing this topic, it was fun, and I miss it. Fortunately, depending on how you look at it, corporate directives are often impossible to predict, so who knows, I could be out there again asking questions very soon. (smile)

Thanks for reading and commenting.

Dan