Comparing Virtual Game Storefronts

Problem

Market Saturation and Design Innovation:

Steam's longstanding dominance (established in 2003) in the digital distribution market raises an intriguing question: could this success lead to complacency and hinder design innovation? Conversely, the Epic Games Store (launched in 2018) entered with a fresh perspective, enticing developers with a lower revenue cut.

User Perception of Design Choices:

To gauge user receptiveness to Epic's design choices, a small research study was conducted. It focused on usability, user experience with commonly desired features, and how each platform met or subverted user expectations. This analysis aimed to understand whether Epic's design decisions resonated with users, ultimately influencing the platform's success.

Solution

This section outlines the methodology employed to understand user behavior and preferences regarding Steam and the Epic Games Store.

Study Participants:

The study involved five individuals with varying levels of experience on both platforms.

Data Collection:

The research spanned three distinct phases, focusing on comparative analysis. Metrics were gathered and analyzed to assess user interaction with each platform. While the full study documentation is available (links provided below), while a brief overview is provided here, the following links have the full reports:

• Wave 1: Performance based metrics, looking deeper into time on task and task success of common items on the platforms.
• Wave 2: Issue Based Metrics, looking deeper into issues faced during the assigned tasks on the platforms.
• Wave 3: Behaviour Based Metrics, looking deeper into general sentiment and efficacy of the platforms for the user.

UX Research

Evaluating Efficiency: User Performance Metrics

The initial focus was on user experience related to ease-of-use.  Five participants completed six pre-defined tasks on each platform with minimal instruction. These tasks aimed to represent common user actions:

• Adding a game to the cart
• Removing a game from the cart
• Utilizing search to find a free game
• Adding multiple free games to the library
• Locating and adding a friend
• Signing out of the account

Task Success and Time Measurement:

Both successful task completion and the time taken to complete each task were recorded. This data was used to compare the user experience in terms of ease-of-use across both platforms for the chosen tasks.

Insights from Average Time on Task:

The first analysis focused on the average time spent & success on each task, providing a preliminary understanding of user efficiency on each platform.

Performance Analysis Reveals Discrepancies:

Our findings revealed a disparity in user performance between the two platforms.  Across the six tasks, users generally:

• Took longer to complete tasks on Steam: This trend suggests potential difficulties in navigating Steam's interface.
• Experienced a higher failure rate within Steam: This indicates that some tasks on Steam might be less intuitive than those on the Epic Games Store.

User Feedback Sheds Light on Difficulties:

To gain deeper insights into these challenges, post-task discussions were conducted with participants. These discussions surfaced the following points regarding Steam's interface:

• Confusing language for adding free games: The wording surrounding adding free games to the library on Steam could be misinterpreted as initiating an immediate download.
• Complexity due to feature richness:  Steam's interface, boasting a wider range of features and interactive elements, might contribute to a steeper learning curve.

These insights will be crucial when exploring design solutions in a later section.

Next Steps: Moving Towards Issue-Based Metrics:

Having established a baseline understanding of user performance on both platforms, the research will now delve into user-reported issues encountered while interacting with each platform.

Uncovering User Issues: Exploring Challenges Through Qualitative Analysis

The second study shifted focus towards user-reported issues faced on each platform. Task completion times and success rates were not the primary focus here.

Exploring Functionality Through Tasks:

Five tasks were assigned to participants:

1. Locate the recent releases section and find a score/rating for a specific game.
2. Implement a filter of choice to refine the game list.
3. Utilize the filter system to find horror games and explore subgenres for a specific game.
4. Locate and adjust parental control settings for adult content.
5. Access content related to a game already in the user's library.

One of the tasks was intentionally designed to be impossible on one platform, aiming to elicit user feedback on this specific limitation.

Think-aloud Method for Capturing User Voice:

Participants were encouraged to utilize the "think-aloud" method, verbalizing their challenges as they encountered them. Speech-to-text software captured this valuable data, offering insights into users' thought processes while navigating the platforms.

Processing Gathered Information:

Unfortunately, due to limitations in the speech-to-text software, only three participants' sessions were successfully transcribed. However, the following visuals and analysis are derived from the conducted usability sessions, offering valuable information about user experiences.

Analysis: Based upon the average amount of issues per task, The Epic Games Store holds issues in functionality that hampers the user experience in a moderate manner. Expectations on navigation and provided information on products is lacking, which caused negativity in participants. Perceptions on what the software should do, when speaking of The Epic Games Store, may have been impacted by the order in which participants were to utilize the software. That Steam was first and has certain functionality and included information may have created an expectation in the user. As for the issues surrounding filtering genres, that was a surprise, further questioning would be required to find out why users had difficulties with the Categories section within Steam, but little issue with the Browse section and filtering within The Epic Games Store.

‍

Analysis: Users found the set of tasks within Steam to be simpler to complete and faced fewer difficulties in doing so compared to the Epic Games Store. Seemingly counter to the hypothesis that claims that the application would overwhelm the user at first, but this question leans towards the user being able to easily comprehend and complete tasks assigned in a more satisfactory manner. This is likely due to being able to complete the tasks assigned universally, users within the EGS had difficulties with reviews and genres not being consistent within games and had to navigate to multiple games at times to find and complete the task assigned. For Steam, all games are consistent in their ratings and page layout.

‍

Analysis: The UMUX-Lite scores indicate that the user experience is greater within Steam than The Epic Games Store, where this difference mainly stems from is quite interesting. Looking at the two questions, ease of use and meeting user requirements, the EGS falls behind within the meeting requirements portion, as ease of use for both applications sits at 28 before calculation. We can directly infer that users find the EGS easy to use but that it fails to meet all of their requirements, linking again to the hypothesis stated that the EGS would be a leaner and more direct experience but lose favour due to lacking features.

User Feedback: Insights from Qualitative Analysis

Steam:

Following the usability sessions, discussions with users revealed valuable insights into the perceived strengths and weaknesses of each platform:

• Adult Content Filter: Participants reported difficulty locating the adult content filter in Steam's menus. It appears this might be a deliberate design choice to limit accessibility. However, some user feedback offered positive aspects:
  • "It keeps people on one page" for information gathering.
  • "The buying experience stays consistent."

The Epic Games Store:

User feedback highlighted several areas for improvement in The Epic Games Store:

• Limited Information: Hovering over games offered minimal details, and information consistency across store pages was lacking.
• Inconsistent Review Access: Finding reviews on game pages was not always straightforward.
• Genre and Sub-Genre Navigation: Users expressed struggles with navigating genres and sub-genres on game pages.
• External Adult Control Settings:  While easier to find than in Steam, activating parental controls opened a separate browser window, disrupting the user experience.

User quotes illustrate these shortcomings:

• "The store page isn't consistent...some have reviews shown one way while they get shown a different way elsewhere."
• "I'd like to see more information from reviewers."
• "Fix the genres and reviews. The end."

Behaviour-Based Metrics: User Exploration and Commentary

This section focuses on user commentary derived from the final study.  Verbatim speech-to-text transcripts from this session are available (link not included here).

Freer Exploration with User Autonomy:

Participants completed four tasks in this final study.  However, these tasks were less rigid, allowing for greater user autonomy in interacting with the platforms. The primary goal was to encourage users to express their overall experiences.  The tasks were:

1. On the main Store page, browse and describe the displayed content.
2. Return to the top of the store page and choose an interesting item to delve deeper into.
3. Analyze and discuss the product page for the chosen item.
4. Navigate to your library and explore game information.

Analysis: Given that the difference in ratios is quite high, there is a substantial difference between the two systems with how they make the user feel. Consideration must be given to the fact that the users are repeat participants who became more aware of issues with platforms in comparison to each other through our sessions.Steam saw many positive comments about the level of information that is available to interact with on the platform, while also getting negative comments about the sheer volume shown on the main store page. Where The Epic Games Store only received one positive comment on the small size of the main store page, and a plethora of negative comments on their lack of consistency and amount of information shown about games.The ratios show that users appreciate that they do not need to leave Steam to find all the information they desire about a game or topic they are interested in, and that The Epic Games Store requires this heavily and has such a low score reinforces this. Overall the ratio indicates that the feel of Steam is an overall positive feeling experience, while for The Epic Games Store it is quite negative.

Analysis: Overall, Steam saw a far greater success rate in meeting the established goals in the combined metrics score. Due to its ease of use and UMUX scoring being quite positive and high compared to the Epic Games Store. The score indicates that the features on Steam meet the comprehensive needs of users far greater than the Epic Games Store.

Analysis and notes: Average time on task was chosen for the Z score analysis instead of Task success due to the success rate being 100% for both platforms. The calculation would not work for The Epic Games Store due to there being no deviation, it gives an error. Reiterating the analysis above, Steam does indeed succeed in combined metrics where the Epic Games Store fails. Quickly going through all metrics collected throughout the first 2 parts of this research process leads to the conclusion that the first 2 hypotheses are mirrored here. Steam does indeed have a higher time on task due to its more complex nature (part 1), and the complex nature presents itself in a plethora of features that users find important (part 2). The Epic Games Store has a lower time on task and higher task success rate due to being much simpler (part 1), but lacks many features that users desire (part 2). So, while the Epic Games Store is far simpler to use, it does not meet the needs of the user and thus fails on a combined metrics analysis.

Testing & Results

Performance Based Metrics

For the issue related to a free games call to action being unclear, depicted below, the button "play game" should state "Install". Paid games on the platform state "add to cart", clearly stating what the button does before you have purchased the game and can play it. This change would reduce confusion about free games on the platform, stating more clearly what the button will do.

Issue Based Metrics

For the issue related to minimal information shows on games icons as at base or on hover, it would be beneficial to add in a review scale, a user review scale that is, and display it based on percentages on a game icon in the store page. Add in 3-4 genre labels on game icons, 1 for the main genre and 2-3 for sub genres. This would give users more information at a glance, and more interactive elements to immediately filter games by genres and sub-genres without needing to go through unnecessary menus.

The above is a "good" example of a game page, and still the review information is not visible. The Epic Games Store should improve readability by forcing consistent layouts and demanding aggregate score links, or a large coloured percentage followed by text which would mimic Steam. This should be coupled with the allowance for user reviews as users stated that the reviews provided were of little importance to them, they prefer a larger quantity of user reviews to draw an overall score from. Also that the reviews are inconsistent in their appearance.

The above is a new opened tab in an internet browser when interacting with the adult controls withing The Epic Games Store. The simple solution is to remove elements, not just the above example, that takes the user out of the application.

Development & Notes

Reflection on the Research Process

This research project proved to be a rewarding experience, solidifying my interest in the field. The process of developing a hypothesis, engaging participants in testing, and fostering open-ended conversations proved both insightful and engaging.  It allowed me to gain valuable user perspectives and understand their unspoken needs, ultimately contributing to the development of a positive long-term user experience.

Looking Forward: Refining Methodology

In future research endeavours, I would aim to expand the participant pool to at least eight, ideally ten individuals. This would strengthen the statistical foundation of the study and provide a broader spectrum of user commentary.