New User Flow A/B Test

Time|2024.02 ~ 2024.12
Role|Research, UX, UI

Overview

Our existing clients SO NICE & PUMA, who build official shopping website on my company's SaaS platform, want to let consumers check the store inventory on the website. This is a customized project and only 2 stores have the feature.

B2C
eLearning

1. Goal

Based on the findings of competitor analysis, deliver a standardized product solution that fits seamlessly with the requirements of both SO NICE and PUMA.

2. How to concatenate 2 systems

Relevant inventory data will be provided through APIs from the client's ERP system.
SO NICE: Integration with ERP via APIs.
PUMA: File transfer via FTP, Our Solution Engineer will convert FTP files into standard API format.

3. Competitors research

Referring to the findings on both TW and international e-commerce platforms, the typical approach for designing an in-store inventory process involves:
1. Displaying an in-store inventory button on the product page
2. Upon clicking, a stock inquiry pop-up emerges, allowing users to query the availability of specific attributes like color and size for a particular SKU at different store locations.

4. User Scenarios & Stories

Motivation: Urgent Needs, Strong Desire, Uncertainty

1. Logistic shipping takes too long, so consumers visit the store to buy the product. (Fast-moving consumer goods, e.g., T&T)

2. While browsing the website, consumers get really excited about a product and want to get it immediately.

3. Unsure about the size, so consumers go to the store to try on. (Clothes & shoes, e.g., LEVIS)

4. To examine the product if the quality matches the price. (High-end products)

5. Consumer journey

6. UI Flow

Design points
✦ The user story is, As a consumer, when the online store is out of stock, I intend to visit a real store for purchase. Hence, I would like to check the inventory in advance.
✦ I didn’t strongly emphasize the entrance button UI. Since "Find in store" is one of the side tasks, "Online shopping" is the main task.
✦ Placed the "Find in store" option below the checkout button. When the checkout button indicates that the SKU is out of stock, consumers can then proceed to click the in-store inventory.

[Desktop] Prototype

[Mobile] Prototype

Test Result

Overview and Objective
The A/B test aimed to increase user bookings, lesson attendance, and purchases by introducing a new onboarding flow that included a welcome screen, learning goal, and hobbies questionnaire.

Key Findings
No statistically significant improvements were observed in key metrics (bookings, attendance, or purchases).
✦ A statistically significant 29% decrease in users reaching the quickbooking portion of the flow was noted, likely due to additional steps in the onboarding process.

Behavioral Insights
Users who completed the additional onboarding steps showed higher engagement and were more likely to attend booked lessons, indicating that the new flow filtered out less motivated users.
✦ The test group had a higher lesson attendance rate (86%) compared to the control group (78%).

Conclusion
The test was concluded with the control flow as the default due to a lack of significant improvement in goaling metrics.
✦ The additional onboarding steps may have value for identifying and engaging more motivated users but at the cost of filtering out less engaged ones.

Recommendations
Testing individual components of the new user flow could be beneficial, but the priority should shift to other experiments and projects.
✦ Consider exploring the personalization of the experience using collected data (e.g., interests or hobbies), though this would require significant effort and scoping.

Next Steps

✦ No immediate action to reintroduce a similar onboarding flow for Engoo.
✦ Reference the findings to address lower lesson attendance rates and explore potential improvements to the overall user experience.