Facilitating an AP World History
Learning Experience for Engram
Usability Evaluation Project, 2021
Project Overview
Engram is a website intended to empower high school students studying AP World History by providing multimedia educational contents that reflect College Board’s curriculum. Priscilla Cancar, our client and the founder of Engram aims to assess whether the website facilitates its mission and to improve the overall usability of the website.
Consisting of four UX researchers, our team conducted moderated remote user tests in order to evaluate the usability of the website of Engram and suggested recommendations to enhance the learning experience for AP World History students.
-
Recruited participants
Moderated user tests
Wrote user test report
Presented to the client
-
Research method: Moderated Remote User Test
Tools: Miro, Google Meet, Figma
-
Jichen Zhu
Sara Almai
Wenjing Wu
-
Oct 20, 2021 - Dec 15, 2021
Evaluation Focus
Through kick-off meeting with our client,
we narrowed down our research focus into following three major points.
General usability of the site
Evaluating the overall readability, navigability and discoverability
Three key pages
Testing out three key pages, the “Introduction”, “Developments” and “People” page
Mission of Engram
Assess whether the Engram website facilitates the mission of Engram
Recruiting Participants
In order to gather the most successful and relevant data, we focused on recruiting participants of two primary user groups of Engram: the AP World History teachers and students. We were able to conduct tests with total of 8 participants that are consisted of 4 teachers and 4 students.
4 AP World History Teachers
Recruited from AP World History Teachers Facebook group, their experience of teaching the course ranged from 1 to 9 years.
4 Former Students
Survey sent via email to Pratt Institute community, students who took the AP World History course between 2008 and 2015 were recruited.
AP World History Teachers Facebook Group
Participant Recruiting Survey
Formulating Tasks
When formulating tasks, we tried to make sure they are clear, concise, natural and unbiased. Also, we limited our user activities to be completed within a specific historical era in order to let the participants to concentrate on the usability side of the site rather than evaluating the historical content itself. Lastly, we asked both teacher and student participants to complete the tasks without relying on their prior historical knowledge.
Scenario: Please imagine that you are a high school student who is taking AP World History. This week, you are learning about the Cold War and for additional understanding of the topic, you are going to use Engram.
Task 1
Take a couple of minutes to browse the homepage. Explain what you think this website is about and what you can do on this website without clicking on anything yet.
Task 2
Find out the three major causes of the Cold War.
Task 3
Find what the Space Race is and what its impact was.
Task 4
You want to watch a video that further explores the death of John F. Kennedy. Please navigate to this to the best of your ability.
Task 5
Find the meaning of Perestroika.
Methodology
We examined the website through moderated remote user test using Google Meet,
which allowed us to observe, record, and analyze the participants as they completed the tasks on the Engram website.
In addition to the tasks, evaluators also prepared pre-test and post-test questionnaires to learn more about
the demographics and thoughts of the participants. These questionnaires included questions about
educational materials used, teaching and learning styles, and feedback, among other things.
Post-test Questionnaire for Teachers
1. What frustrated you most about this website and what would you change?
2. What did you like about the website?
3. How likely are you to recommend this website to your students on a scale of 0-10 and why?
Post-test Questionnaire for Students
1. What frustrated you most about this website and what would you change?
2. What did you like about the website?
3. How likely is it that you would have used this site for the course on a scale of 0-10 and why?
Pre-test Questionnaire for Teachers
1. How long have you been teaching AP World History?
2. What kind of educational materials do you use for the class?
3. Have you ever used online materials, e.g., YouTube or Wikipedia?
Pre-test Questionnaire for Students
1. What kind of educational materials did you use during the class?
2. Have you ever used online materials?
3. What was the most frustrating part of studying AP World History?
Moderated User Test on Google Meet
User Screen Sharing
Data Analysis
Once all the user test sessions were completed, we collected the data using affinity mapping.
The usability issues were then consolidated into a single table, providing a consistent document of qualitative and quantitative data.
This allowed us to better assess patterns and trends and effectively pinpoint findings to improve overall usability.
In order to prioritize what usability findings were most important, we rated the severity of the findings based on a 0-4 point scale.
Affinity Mapping of User Test Data
Table of Usability Issues
Teacher Participants Demographics
Former Student Participants Demographics
Post-test Questionnaire Responses
Overall Feedback
7 out of 8 participants were able to complete all of the tasks successfully.
The average rating given by students was 5.5/10 and the average rating given by teachers was 7/10 for an overall average of 6.25/10.
19 usability issues of various severity ratings were identified. Two of them were given a severity rating of 4 and are vital to fix.
3 Major Issues
Among various usability issues, we chose three key findings that were most vital for improvement.
1. Navigation: Participants felt confused about the ordering of sections on the historical event selection page, the “People” page and the “Terms” page.
2. Layout: The unclear structure led participants to spend unnecessary time looking at peripheral materials while trying to find the answer. Also, current font size could be unfriendly to users with weaker eyesight.
3. Interactivity: Participants found difficulty in interacting with the “Developments” page and misunderstood the function of the world map and timeline bar.
Issue 1.
Navigation
Findings
Many participants noted that the overall aesthetic of the Engram website is appealing and has a clear distribution of operation modules which enables easy navigation. However, some participants pointed out that, first, the main page for each topic has the subtopics arranged in chronological order, but the historical event selection page does not have an identifiable order (Annotation 1). Some participants said the order of this page was confusing for them.
Second, the ”People” page and “Terms” page also do not have an identifiable order (Annotation 2).
Lastly, there is no search function for the website (Annotation 3). Our participants commented that this makes it hard to find specific contents, especially for students who are new to learning about a certain topic.
Recommendations
In order to help users out, we compiled a couple of recommendations. On the historical event selection page, we recommend adding the starting year and ending year of each historical event, as well as arranging them in chronological order (Annotation 1). This change could also make it easier for users to distinguish between historical events which could increase understanding.
Secondly, we recommend arranging the content in alphabetical order on the "People" page and "Terms" page, which could help users find relevant information more quickly (Annotation 2).
Lastly, we recommend adding a search function to the navigation bar at the top of the page, so that users can use keywords to find specific content quickly and accurately at their convenience (Annotation 3).
Issue 2.
Layout
Findings
On the “Introduction” page, there is a lot of helpful material available, such as a world map displaying relevant regions and a paragraph detailing lasting impacts. Many participants found the page convenient. However, the unclear content structure led participants to spend unnecessary time looking at peripheral materials to find the information they were looking for (Annotation 1). One participant mentioned, “My students are not going to read that. The information is not big enough.”
Additionally, a couple participants pointed out that the font size used was too small and therefore potentially unfriendly to users with weaker eyesight. The font size for the body paragraph of Engram was 12px (Annotation 2).
Recommendations
In order to improve the clarity and accessibility of the content structure, firstly, we recommend using subheadings and bullet points, which can help users quickly pinpoint information as well as improve overall comprehensibility (Annotation 1). One participant who is a current AP World History teacher asserted, “The more broken down, the better.” Secondly, we recommend enlarging the font size to 18px to improve readability (Annotation 2). Lastly, we recommend allowing the enlargement of relevant images on the website, enabling users to click to see a larger version if needed (Annotation 3).
Issue 3.
Interactivity
Findings
Interactivity issue was identified on the “Developments” page.
The “Developments” page consists of a world map, a timeline bar, and a carousel in order to feature important events that took place during each era. Participants who were current teachers gave positive feedback on the page and mentioned that it could help students retain information by providing clear visuals.
However, the participants had difficulty navigating the different content offered on the page. This seemed to be because the highlight effect on the map when hovering gives a false indication of interactivity. Because of this effect, most of the participants assumed that the map was clickable.
In addition, the arrows on the carousel, which is the interaction controller to switch between the different developments, are not discoverable because they only appear when hovering.
Recommendations
In order to enhance the interactivity of the “Developments” page, we first recommend removing the highlight effect that occurs when hovering over different regions of the world map (Annotation 1).
Secondly, we recommend that the arrows of the carousel appear on the screen at all times and not just when being hovered over. This would increase discoverability and be an indication to users that they can interact with this feature in this capacity (Annotation 2).
Lastly, we recommend adding a textbox including simple instructions for how to properly use the “Developments” page (Annotation 3).
Presenting to the Client
“I have been working on Engram for too long and was too close to see it from a new perspective. I could tell a lot of care was put towards considering what our users want. I cannot wait to read the entire report!
- Priscilla Cancer, the owner and founder of Engram
Our client was very impressed with our work and especially was very excited to hear the voices of the real users.
She was also curious about our personal experiences throughout the entire process of our project, which was shared after the presentation.
The client was very looking forward to read the user test report that was sent right after the presentation
and hopes to implement all the recommendations that we have suggested.