Evaluating Trifacta Wrangler

Academic Research & Usability Testing

Project Overview

With more and more companies becoming data driven, so-called casual users that work in marketing and sales departments depend on software to help them process and analyse data without needing technical know-how or help from IT. However, studies reveal that there is a need for user centered data preparation tools that better meet casual user's needs for self-service.
‍
This project adresses this issue and showcases skills relating to usability engineering and academic research in general.

The thesis project was conducted over a period of 9 months and supported by Deploy Teq who made it possible to access real participants adding to the study result's credibility.

Methodology

To identify usability problems a usability test was conducted with 8 participants. The concurrent think aloud method was chosen to collect qualitative data. The advantage compared to the retrospective think aloud is that subjects do not report false memory.
The user action framework (UAF) was chosen to classify usability problems. It is derived from Donald Norman's theory of action and additionally assigns a severity rating to each problem based on their frequency of occurence and influence on task completion. The other usablility classification schemes reviewed did not include severity ratings or had other shortcomings.

Each of the 8 subjects was given the same 3 data structuring tasks in Trifacta Wrangler with 15 minutes to complete them on a demo data set. In addition they filled in a survey about their technical skill.

Figure 1: A visualization of the methodology applied in the research study.

Results
It was found that most usability problems belonged to the planning and translation phase of the UAF interaction cycle. Contrary to previous research, participants’ Excel skill influenced their capability to plan interactions negatively. The System Usability Scale revealed that this might be related to learnability as a usability criterium. In line with previous research, one participant with prior SQL experience recovered from the most severe planningissue due to their knowledge of programming concepts.

It can be said that to improve the self-service level of casual users, understanding their planning of data structuring tasks is crucial. Further research is needed to verify these findings by identifying tools and technologies used by a larger sample of casual users and having them perform data structuring tasks in various market-leading data preparation tools.

Access the complete thesis document

Utrecht University

Thesis Msc. Human-Computer Interaction

Feb 2022 — Nov 2022

Figure 2: A comparison between the complete UAF usability classification tree (left) and the classifications in the usablity problem inventory (right) relating to casual user's interactions with Trifacta Wrangler. It can be seen that the Planning and Translation issues occurred most frequently during the usability tests.

Quotes taken from the coded transcripts illustrating the importance of user's experience with Excel when using new data preparation tools.

Figure 3: Quotes taken from the coded transcripts illustrating the importance of user's experience with Excel when using new data preparation tools. Based on Jakob Nielsen's law of UX, designers should incooporate interaction patterns from Excel in future BI applications. More on Jakob's law

The severity table shows which usability problems should be addressed most urgently. P3a relates to casual user's with less programming knowledge not knowing that string-values must be written in quotes.

Figure 4: The severity table shows which usability problems should be addressed most urgently. P3a relates to casual user's with less programming knowledge not knowing that string-values must be written in quotes. Figure 5 shows how the participant with SQL experience recovered from the same error.

The think aloud transcript revealed that SQL knowledge helped user's to recover from the most severe usability problem that would otherwiselead to task failure. Improving the feedback messages for casual users would be a low-cost fix and would greatly improve the usability of Trifacta Wrangler.

Figure 5: The think aloud transcript revealed that SQL knowledge helped user's to recover from the most severe usability problem that would otherwiselead to task failure. Improving the feedback messages for casual users would be a low-cost fix and would greatly improve the usability of Trifacta Wrangler.