CSC 486 PASS Usability Evaluation Team - Final Report <title> <link rel="stylesheet" type="text/css" href="style.css" media="screen"/> <link rel="stylesheet" href="thickbox.css" media="screen" type="text/css" /> <script type="text/javascript" src="jquery.js"></script> <script type="text/javascript" src="thickbox.js"></script> <style type="text/css"> p { text-indent: 1cm } </style> </head> <body> <div id="header"> <div class="center_wrapper"> <div class="clearer"> </div> <div id="site_title"> <br /> <h1><a href="#">Final Report: <span>PASS Usability Evaluation Team</span></a></h1> <p style="text-indent:0em;">By Justin Appler, Dac Collins, Ryan Lange, and Björn Lindfors</p> </div> </div> </div> <div id="main_wrapper_outer"> <div id="main_wrapper_inner"> <div class="center_wrapper"> <div class="left" id="main"> <div id="main_content"> <div class="post"> <script type="text/javascript" src="https://media.dreamhost.com/swfobject.js"></script> <h2><a name="intro"></a>1. Introduction</h2> <p> For the past several years, students at Cal Poly have been using a new system called Plan a Student Schedule, or PASS, to search for classes and create schedules. While some would argue that PASS is a great improvement over the previous system, there are still issues for many users trying to build their schedule. </p> <p> Instead of building and evaluating a new piece of software, we decided to look at existing issues with the PASS system. In this evaluation, we decided to focus on two main groups of users. The first, New Users, are users that have never registered for classes before. While we used engineers and employees at Intuit for our testing, the 'New Users' are ideally Freshman who are asked, often without support, to use the system for the first time. The second group, which we call 'Experienced Users', are users that have already registered using the PASS system before and are now looking to increase the speed at which they accomplish their goals. These groups make up individuals who are trying to do what we have found to be the most difficult aspects PASS: using it for the first time and using it quickly and effectively after that. </p> <p> Despite great improvement, we will show that PASS is still in dire need of help. Issues ranging from unclear workflows to overly textual data representation exist within PASS and need to be resolved. After analyzing these issues, we will provide not only solutions to remedy them, but a few prototypes for entirely new interfaces. </p> <h2><a name="motiv"></a>2. Motivation & Purpose</h2> <h3>2.1 Relevance</h3> <p> From our own experiences and from the discussions we've had from numerous other students we know that our registration system is in need of some changes both is usability and functionality. Many students have shared in similar experiences that have resulted in hours of time wasted in finding a decent schedule or thinking that they are registering for their classes but later realize that they did it wrong, resulting in not enrolling in any classes. These issues are both very frustrating and unproductive. It is for these reasons that we decided to take a look at this issue to see if we could come up with a better solution to provide students with an easy procedure to find, register, and enroll in their desired courses. </p> <h3>2.2 Motivation</h3> <p> Registration time is not fun and with our current registration system it can be very frustrating and tedious as well. This is what drives our motivation to evaluate PASS/CPREG. We wanted to come up with an interface that would make it very simple to do various tasks in a short amount of time so that students would not dread registration day. Students need to focus on doing their work so that they can succeed in their courses. Instead during registration time the students are spending hours trying to find courses that fit certain requirements and then making sure that their courses will not have any time conflicts. </p> <h3>2.3 Purpose</h3> <p> The purpose of our research and analysis of this issue is to provide a user interface that will reduce the headache of registration time by allowing students to create optimized schedules in little time and effort. </p> <h2><a name="back"></a>3. Background</h2> <p> The registration period is a stressful time of the quarter for any student; Cal Poly students are no different. There are computer systems in place for planning and registering for classes. The system that is used to lay out your classes for the next quarter will be covered in this document. </p> <p> PASS or “Plan A Student Schedule” is a web application used by students to check class availability and possible scheduling combinations. Classes are sorted by department for better overview. When you select classes they are put in a “shopping cart”; when the user proceeds to the next screen the system will present possible scheduling combinations based on the classes in your shopping cart. Advancing yet another screen allows the user to view or save his/her newly created schedule. </p> <p> However, the user can not apply for his/her courses through PASS alone. Another system is needed: CPReg. It will not be covered in great detail, but what will be covered in some details is how users cross-reference systems, how they move information from one system to another. </p> <p> The primary focus of this paper will be to evaluate the usability of PASS; the secondary focus to assess how users move information from PASS to CPReg. </p> <h2><a name="init"></a>4. Initial Impressions</h2> <h3>4.1 Ryan</h3> <p> Before even enrolling in this course I remember thinking of ways that I could improve the system. When we decided to evaluate PASS/CPREG I was exited because I would now have the opportunity to do more. My initial concern was the the fact that there are 3 applications that are needed to find and enroll in courses. The first is the Cal Poly website, where students need to find the courses that meet certain requirements such as the GWR, USCP, and section in their G.E. The second is PASS and then finally the third is CPREG. This can be very confusing for both new and continuing students, and is very bad design. Another concern that I had were the lack of filtering options to search for classes. Basically the only way we have to find a class is through department. I have spent hours trying to find the right courses I need and then trying to fit them in a schedule without a conflicting time constraint. Also, there are not any options to search by professor, day, or time. I feel that these improvements would improve the usability of the system tremendously. </p> <h3>4.2 Björn</h3> <p> When I was confronted with Cal Poly’s computer system I was appalled to say the least. The system in place at my home university, Chalmers back home in Sweden is not perfect. It has it’s flaws. But at least it is all integrated into the same system; not spread out over several systems that has no possibility of talking to each other. I never had much trouble using PASS, but there is definitely room for improvement on the usability aspects of the system. For example: Why is it not possible to filter and search for classes based on your own criteria? The only possible way of filtering classes today is by department; which is fortunately not a bad way to do so. An iTunes-like interface would work well for PASS in my opinion; with a menu bar on the left, class listing in the main, right column. Filtering options would be available to the user so that he could sort the classes based on his own criteria; a free form search would also be supplied. </p> <h3>4.3 Justin</h3> <p> Prior to PASS, registering for classes was generally a nightmare. It required looking back and forth at the class catalogue, looking up approved general education courses, checking individual department websites, and spending time and effort without getting good results. When PASS arrived, a collective sigh of relief could be heard. It made huge improvements on the existing platform and was generally quite easy to use. That said, there are still a number of issues that make PASS difficult to use. The first I noticed was the inability to search by Professor. At some point in one's college career, they realize that sometimes the Professor can have a larger impact on success in a class than most other factors. As such, many students will take classes simply because a talented Professor is teaching them. Features like this are simple and sensible yet missing completely. I assumed that during our testing we would find things like this to be the most common issue, but that overall people would be happy using PASS. </p> <h3>4.4 Dac</h3> <p> I come to this project from the unique position of having worked on a PASS replacement system in the past. Although PASS is certainly an improvement of Cal Poly's previous registration system, it suffers several major design flaws. First and foremost is the wizard-like design. It is simple to use for new users, but forces one to go back and forth between steps over and over to test different arrangements of classes. Moreover, since PASS doesn't provide any information about which classes are causing conflicts, this process quickly degrades to trial and error, which can be frustrating and time consuming. Second, almost every quarter, I want to take Class A or Class B, but not both. PASS lacks this feature. Lastly, of course, I'm repeatedly frustrated by having to jump back and forth between PASS and CPREG to add classes. This is even more annoying when trying to crash classes, because I have to use CPReg to get up to date enrollment info, then go back to PASS to actually see class times, etc. Overall, PASS was a good start, but it needs work. </p> <h2><a name="new"></a>5. New User Impressions</h2> <p> While most users have used PASS before at some point, there are a group of freshman each year that are using it for the very first time. These users generally have never seen it before and are unfamiliar with the registration process. While it is important for us to see how experienced Cal Poly students can utilize PASS, it is equally important that we see how someone using PASS for the first time reacts to the system. To do so, we had 2 Intuit Engineers (which would represent maybe the 'best equipped' new users to the PASS system) complete various tasks exactly the same as those provided to the experienced users. </p> <b>Filtering</b> <p> There is no functionality to list courses by professor, time, or day. </p> <b>Workflow</b> <p> The linear workflow often gets users confused on where they are in the process. Users don't understand why they need to move 'forward' to see possible schedules based on current selections. They also don't understand why they need to move forward to see the contents of a given schedule. </p> <b>Blocking</b> <p> If a student has work everyday at the same time or just does not want to have classes during a certain period of the day there should be a feature that blocks all the courses that are offered during that period, in order to reduces the number of choices and time to review each course. </p> <b>Placement of Relevant Information</b> <p> Relevant information about classes, such as the number of units, can't be seen until the class is selected. This is one situation where the user is expected to know more about the class than they should. </p> <b>Grouping</b> <p> In the CSC/CPE/SE department, for example, classes for each major are spread out between the CSC and CPE departments. Grouping the classes by Major would be more helpful to the students trying to find CSC101 when there is no CSC101 listed. </p> <h2><a name="exp"></a>6. Experienced User Impressions</h2> <p> The experienced user impressions were gathered using <a href="composite.pdf">this</a> form. The summary of those results are below. </p> <b>1. Tell us your worst registration story.</b> <p> Most users reported stories related to lack of class availability rather than system failure. Exceptions are problems with classes being dropped, particularly in that CPReg clears waitlisted classes. </p> <b>2. What criteria do you use for selected courses.</b> <p> The most common criteria, naturally, was the student's major requirements (i.e the classes they "need" to take.) On top of that, most students used time first and professor second, though that was reversed for one student who relied heavily on Polyratings. </p> <b>3. What features of PASS do you like most?</b> <p> Users really liked that they can see quick previews of their schedules in a mini-calender. Users also liked the schedule sorting options, particularly "Least/Most ... class bocks" and "Fewest days in class." The new View on Map feature was unknown to many users with most thinking it would be most useful to freshman. </p> <b>4. Do you have any improvement suggestions?</b> <p> Most of the suggestions were related to Cal Poly's registration system in general rather than PASS in particular. One user proposed a rotation in which students can add one class at a time, per rotation. He also suggested that some form of bidding or weighting system could be explored. The hope was to make registration more fair and less all or nothing, a problem several users reported. A different user suggesting tying PASS to the popular professor rating website PolyRatings.com, a website every student at least mentioned. Two users suggested displaying the unit count next to a course in the first PASS screen, a simple change we highly recommend. </p> <b>5. After using PASS, how do you enroll in classes on CPReg?</b> <p> Somewhat surprisingly, most users still wrote down all their classes on paper before transferring them to CPReg. Reasons given: Always trust paper, afraid of high server load, and bugs with having both apps open. This contributed to an overall lack of trust in PASS/CPReg. </p> <b>6. What is your emotional state when using PASS?</b> <p> Feelings ranged from excited to nervous, stemming from the idea of new class and the possibility of not getting them, respecitively. One user said that she knew PASS has caused her frustration in the past, but couldn't place where. </p> <b>7. On a scale of 1 to 10, rate your overall user experience.</b> <p> Average: 7.1. Not bad at all for a relatively young system. Users were able to get classes more or less without problems and simply learned their way around PASS's weaknesses. One user did say their would definitely be ways to improve it. </p> <h2><a name="overhaul"></a>7. UI Overhaul</h2> <p> Each of these prototypes attempts to address the issues described in the sections above. Most include want users wanted most: the ability to 'search' for classes like when using a modern search engine. </p> <h3>7.1 Prototypes</h3> <b>Click on any of the prototypes below to see a full size image</b><br /><br /> <a href="mockups001.jpg" class="thickbox"><img width="255" height="350" src="mockups001.jpg" /></a><br /><br /> <a href="mockups002.jpg" class="thickbox"><img width="350" height="255" src="mockups002.jpg" /></a><br /><br /> <a href="mockups003.jpg" class="thickbox"><img width="350" height="255" src="mockups003.jpg" /></a><br /><br /> <a href="mockups004.jpg" class="thickbox"><img width="350" height="255" src="mockups004.jpg" /></a><br /><br /> <a href="mockups005.jpg" class="thickbox"><img width="255" height="350" src="mockups005.jpg" /></a><br /><br /> <a href="mockups006.jpg" class="thickbox"><img width="298" height="232" src="mockups006.jpg" /></a><br /><br /> <h2><a name="conc"></a>8. Conclusions</h2> <p> We initially wanted to develop an interface to replace the PASS. As software power users, the slow, wizard-like process frustrated us and we wanted a radical overhaul. This is reflected in our UI mockups. However, based on the information we gained over this quarter, we suggest that PASS is actually serving students adequately and that incremental improvement would be more beneficial. Below we list changes that we recommend, in order of importance. </p> <ol> <li> Integrate with CPReg<br /> Obvious. Reduces complexity and time and eliminates CPReg errors. Increases trust in the PASS as the registration system instead of users seeing it merely as a way to view available classes. A simple "Enroll in Schedule" button would be great. </li> <li> Include a "Units" field on in Step 1<br /> Simple to implement and mentioned by several students </li> <li> Allow saved schedules to be "reloaded" so they can be edited<br /> Many students do not even save schedules. Without the ability to re open and continue editing them, saved schedules have little benefit over a piece of paper. Should be easy to implement. </li> <li> When a schedule cannot be found, list the overlapping classes.<br /> Eliminates the guess and check. Idea: still generate schedules and show overlapping classes in red on the mini view and in the list. </li> <li> Add Filtering and/or Search<br /> Some classes may not belong to an obvious department and finding them is slow (eg Computer Science classes are spread between CPE and CSC). No way to look up a professor at all. <ol> <li> By time <br /> Most popular criteria </li> <li> By professor<br /> Also popular, especially among Polyratings users </li> <li> Keyword search </li> </ol> </li> <li> Integrate steps 2 and 3 by assuming users want all sections checked.<br /> Every user interviewed but one checked every box in Step 2. The other user used Step to pick his preferred professor (see below). </li> <li> Show professors and location of courses in Step 3. <br /> Eliminates the need for Step 2. </li> </ol> <p> On top of these changes, we suggest a few new features that PASS implementors should consider. </p> <ol> <li> Add some OR relationship between classes, possibly through grouping. <br /> Users may want to take ANY GE area B3. Or, may want to take three of the following four classes. </li> <li> After implementing 6 and 7 above, consider further condensing the number of steps<br /> The 1...2...3...4 metaphor makes things easy for new users, but most users (other than first quarter freshman) are not new. So the priority should be given to speedy efficient schedule generation for experienced users. </li> <li> For the reason above, don't focus on the Map feature.<br /> </li> <li> Integrate PASS with departmental course requirements and PeopleSoft's progress report.<br /> For every user, the classes they needed to take that quarter (eg as per their flowchart) was the biggest criteria for class selection. Not having to jump back and forth between different pages and even PDF's would save tons of time. Being able to see what classes one has already taken would eliminate the possiblity of taking the same class or a class that meets the same requirements twice. </li> </ol> <p> Some of the changes are simple, some more difficult, but each one would greatly improve the overall registration system. Implementors should be encouraged by the 7.1 rating that users gave the system and by the positive comments with respect to previous enrollment systems. This usability evaluation yielded good, sometimes surprising results, reminding us yet again that software must be tested again and again with its users for best results. </p> <h2><a name="footage"></a>9. Evaluation Screen Capture Footage</h2> <h3>9.1 New User Evaluation Footage</h3> <p> <div id="player1"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','576','432','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','432'); so.addVariable('width','576'); so.addVariable('file','http://pelicansoiree.com/486/part1.flv'); so.write('player1'); </script> </p> <p> <div id="player2"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','576','432','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','432'); so.addVariable('width','576'); so.addVariable('file','http://pelicansoiree.com/486/part2.flv'); so.write('player2'); </script> </p> <h3>9.2 Experienced User Exercises</h3> <b>Participant 1</b> <p> <div id="player31"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Becca.FLV'); so.write('player31'); </script> <br /><br /> <a href="user1.swf">Click here to see the screen capture footage</a> </p> <b>Participant 2</b> <p> <div id="player32"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Cool_Hair.FLV'); so.write('player32'); </script> <br /><br /> <a href="user2.swf">Click here to see the screen capture footage</a> </p> <b>Participant 3</b> <p> <div id="player33"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Creepy.FLV'); so.write('player33'); </script> <br /><br /> <a href="user3.swf">Click here to see the screen capture footage</a> </p> <b>Participant 4</b> <p> <div id="player34"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Hillary.FLV'); so.write('player34'); </script> <br /><br /> <a href="user4.swf">Click here to see the screen capture footage</a> </p> <b>Participant 5</b> <p> <div id="player35"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Pair.FLV'); so.write('player35'); </script> <br /><br /> <a href="user5.swf">Click here to see the screen capture footage</a> </p> <h3>9.3 Experienced User Questions</h3> <b>Question 1: Tell us your worst registration story.</b> <p> <div id="player3"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Question_1.FLV'); so.write('player3'); </script> </p> <b>Question 2: What criteria do you use for selected courses?</b> <p> <div id="player4"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Question_2.FLV'); so.write('player4'); </script> </p> <b>Question 3: What features of PASS do you like most?</b> <p> <div id="player5"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Question_3.FLV'); so.write('player5'); </script> </p> <b>Question 4: Do you have any improvement suggestions?</b> <p> <div id="player6"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Question_4.FLV'); so.write('player6'); </script> </p> <b>Question 5: After using PASS, how do you enroll in classes on CPReg?</b> <p> <div id="player7"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Question_5.FLV'); so.write('player7'); </script> </p> <b>Question 6: What is your emotional state when using PASS?</b> <p> <div id="player8"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Question_6.FLV'); so.write('player8'); </script> </p> <b>Question 7: On a scale of 1 to 10, rate your overall user experience.</b> <p> <div id="player9"> <a href="http://www.macromedia.com/go/getflashplayer">Get the Flash Player</a> to see this player. </div> <script type="text/javascript"> var so = new SWFObject('https://media.dreamhost.com/mediaplayer.swf','mpl','480','320','8'); so.addParam('allowscriptaccess','always'); so.addParam('allowfullscreen','true'); so.addVariable('height','320'); so.addVariable('width','480'); so.addVariable('file','http://486vids.pelicansoiree.com/486_Project_Question_7.FLV'); so.write('player9'); </script> </p> </div> </div> </div> <div class="right" id="sidebar"> <div id="sidebar_content"> <div class="box"> <div class="box_title">Table of Contents</div> <div class="box_content"> <ul> <li>1. <a href="#intro">Introduction</a></li> <li>2. <a href="#motiv">Motivation/Purpose</a></li> <li>3. <a href="#back">Background</a></li> <li>4. <a href="#init">Initial Impressions</a></li> <li>5. <a href="#new">New User Impressions (Intuit Usability Evaluations)</a></li> <li>6. <a href="#exp">Experienced User Impressions (Library Usability Evals)</a></li> <li>7. <a href="#overhaul">UI Overhaul</a></li> <li>8. <a href="#conc">Conclusions</a></li> <li>9. <a href="#footage">Evaluation Footage</a></li> </ul> </div> </div> </div> </div> <div class="clearer"> </div> </div> </div> </div> <div id="footer"> <div class="center_wrapper"> <div class="left"> CSC486 PASS Usability Evaluation Team </div> <div class="right"> <a href="http://templates.arcsin.se/">Website template</a> by <a href="http://arcsin.se/">Arcsin</a> </div> <div class="clearer"> </div> </div> </div> </body> </html>