5 Considerations for User Experience (UX) Researchers running remote Virtual Reality (VR) studies
Takeaways from a participant perspective.
I started playing Beat Saber last month on my partner’s Oculus Rift and beat my first song on hard (League of Legend’s KDA) after hours of frustration. I then joked it was probably possible to pay off the expensive VR headset strictly through remuneration from remote VR studies at universities. A couple of weeks later, I had participated in 6 VR studies of varying lengths and the headset was ~46% paid off in Amazon giftcards. Through this curious endeavour, I intimately developed an understanding of the remote participant experience where I often found myself as an ad-hoc research assistant.
The main difference between in-lab and remote studies is that the researcher loses control of the physical space, and they can no longer point to buttons of interest, direct the participant by tapping their arm, or personally adjust hardware. Instead, they must rely on the participant to self-direct and the participant inevitably becomes the research assistant, learning vocabulary, calibrating set-up, and managing collected data. The researcher must prepare the participant to fill that role.
Here are 5 suggestions for User Experience (UX) and Human-Computer Interaction (HCI) researchers who are transitioning their in-lab VR study to a remote online format.
1. Follow a script for VR literacy
Define UI terms. Raycast? Hitbox? Modal? These terms are common to designers yet unfamiliar to study participants unacquainted with the area. Even the word “experience” in “game experience” can mean different things without context. Although it may be time-intensive to define every term for participants, ask the participant to define key terms before repeatedly using it with your judgement.
Define inputs. List through the touch controller buttons your participant will be using in the study with them, so they will know which button you are referring to during the study. Some buttons have different names, like “palm trigger,” “middle finger trigger,” ‘Primary Hand Trigger,” and “grip button” which all to refer to the same button.
If the buttons are labelled (icons, or call-out lines) within the VR build your participant is using, make sure it is the same as how you would verbally refer to the buttons. Instead of relying off conversational know-how, this is where a script will come in handy even just to keep you from referring to the “B Button” as “Button Two.” Following a script can also encourage the participant to exercise patience and listen attentively to instruction.
2. Consistently calibrate hardware
Supervise Set-up. Before each study, instruct the participant to calibrate their VR space such as by remapping the play space and making sure that their height is correctly entered. If the participant is in a shared space, someone else may have used the headset before the study and the participant may need to re-calibrate it for their body. Allot time for calibration in your study or maybe even program in adjustment tools into your build, such as a slider for virtual floor height.
Supervise Start-up. If a player could send you a photo of their physcial space prior to the study, you could get a sense of context in order to navigate them during the experiments. For example you could say “Move closer to your couch.” Also, request the player to start the build from their virtual desktop within their VR environment. If your participant was seated at a desktop upon start-up, the build may take your participant’s seated height as their real height and create an environment at the desktop rather than in the participant’s intended play space. Ask the participant to place their controllers on the floor and see if the real floor is at the same height as the virtual floor.
If study repeatability is your goal, consider creating a set-up/start-up manual for participants to follow.
Some common set-up errors I’ve encountered are:
· Improperly angled cameras
· Cameras obscured by objects
· Height not adjusted to the participant’s height
· Headset not put on properly
· Battery low in controllers
· Inadequate space to extend arms
3. State expectations for software
State computer requirements. University lab computers are rigorous machines compared to most personal computers. In a remote experiment, you may need the participant to download a build, run the build, be on a videocall with you in real-time, record data including video data, and maybe open some links for surveys and consent forms. Make these steps known to participants at the point of recruitment, including the size of the build. During the study is not the time to realize that their computer or network is unable to support the study.
Familiarize with their headset. Prior to the study, ask the participant which headset they will be using so you can prepare to troubleshoot potential issues together. If problems arise, you may need to give specific button-by-button directions to complete it. Don’t assume that participants know how to set up their VR headset independently as (like in my case) the headset may have been borrowed. Ask them: “Would you like instructions on how to reset view?”
If this level of VR assistance is not practical, VR expertise could be screened for in recruitment surveys with questions like “Describe the steps to reset the view on your headset.”
4. Provide instructions for videocalls
Instruct screensharing. If you are communicating with or observing your participant during the study through a videocall, make sure your participant has the videocall program’s client downloaded and that screen sharing is enabled. Browser-based versions of many videocall services like Zoom can be limiting. Also, specify to participants on how to screenshare such as whether they would be sharing their full screen or just the VR build’s application. Give participants adequate time to close their personal tabs prior to screensharing and allow them to let others in shared spaces know they are on a call.
Organize speakers and mics. With a VR headset plugged in and a videocall in the background, there will be many headphones and mics available. The participant should use the VR headset as the videocall’s audio in/outputs during the experiment but during parts outside of VR such as during surveys or interviews, they should be reminded to switch to their system audio in/outputs as the VR headset would not be worn.
Participants can be residing anywhere in remote studies, thus you should also be aware of certain website restrictions in the regions your participants are from (i.e. Google applications are not accessible in China). If this poses as a problem, build it into your screening survey during recruitment.
5. Follow-up after data collection
If your remote study requires the participant to send you any collected data files after the experiment, set aside time for them to send it and make sure the .zip file or .rar file can be opened. In the case the file is too large for e-mail, prepare a cloud storage option (i.e. Google Drive or Dropbox) where the participant can easily upload the files. Ideally, the files would be sent over and confirmed before the study session ends, but if it is not practical, have the participant confirm that they know which files to send over and then follow-up with them after you have received the files. Request participants to keep the needed data files for a week in case they need to resend them.
5.1 (BONUS) Have a contingency plan!
If a study cannot be run at the agreed upon time, it is ok to reschedule the study to a later time. Even if the study has gone through part-way, ask the participant if you could meet again a day later if it makes sense to your study design. While designing your study, determine protocol in the case of a crash, such as how remuneration will be handled and what the next steps would be. If there are study parts independent of each other, can the study be split up over days? Is it still beneficial for participants to give partial response of data?
The time spent troubleshooting a VR environment may not directly contribute to a study, but you are still working with participants’ time, which is ethically perceived as an economic risk (i.e. time spent not playing Beat Saber). A 15 minute experiment could actually take 30 minutes of the participant’s time with set up and this time should be considered in study design and made aware during recruitment.
While time is valued during times of juggled responsibilities during quarantine, if you have access to a VR headset and time of your own, it is also definitely worth participating in fellow researchers’ studies! Through my conversations as a research participant, I learned that it can be quite hard right now to recruit participants with VR access outside of closed labs.
