I haven't tried Loop11 yet, but it looks like a convenient way to set up remote, user-administered tests for mobile devices.

Jakob Nielsen (www.useit.com) just posted an article about their mobile usability testing: http://www.useit.com/alertbox/mobile-usability.html They used twitter to have users enter what they did on their phone immediately during the day and then did a follow up questionairre each night. You could probably ask your test subjects to do the same and to make sure they gave the location that they were when they entered it. You would obviously need to instruct them on the level of detail here and would try to keep it to a minimum and easily entered on a mobile phone. Scott

First question asked: "How can we collect user feedback in a reliable way without observing our users? What are current field methods used in the mobile world?" Keyword here is 'reliable'; this says to me we are trying to get feedback on a daily or some routine basis. What if a user forgets to log in to a Wiki routinely, or wherever feedback is meant to be posted? To maintain some regularity, I would recommend utilizing some form of Push Technology (PNS on iPhones) that serves as a reminder to the user on a pre-set basis when the game is launched or run. (This would be for the Beta only -- obviously, this would be a distraction in the final version!) PNS allows server side notification alerts to be set up that don’t require mobile applications to stay running in the background just to update users of the external events they track. Question 2: "How can we capture contextual information on the situations users play the game and the features they choose to play?" Push and pull technology could be utilized at preset intervals to reach out to the smart/mobile phone to capture information such as features played, how long feature set was played. Obviously, all of this does require some ability to capture information while the game is active, perhaps by writing out data to XML/data file with the application itself. Third Question: "Having users annotate their thoughts/comments on a form on a daily basis versus recalling all thoughts at the end of the experience?" I think the answer to Number 1 would suffice. Another option would be to build a simple text system where a user could input info and click Submit to send directly to a database. See "Distant Shore" in the iTunes store for a working example of this, where simple text messages are sent to random users; these messages end up on another user's beach to which they can respond to the sender if desired. For this application, simple text messages could be sent to your server's own database, or wherever you would maintain user comments / thoughts.

Find the closest game or app that is mobile and location-based. Maybe Loopt or something. See how users interact with that (ethnography or monitoring their posts). Then apply anything you learn to your game. OR Just get your game working in the iPhone app store or wherever you plan to release. Treat it like a beta and make it free. Offer incentives like lifetime subscription for free or Amazon gift cards for the most updates/use. Then just sit back and observe what the users do with it.

I would recommend a multi-method approach. 1. find a handful of users to shadow and observe. This will uncover a large number of findings with low participant numbers. If desired, follow up with small interviews with a few users, including some who were not observed. 2. Questionnaires can be easy to administer and very informative. Formulate survey instrument according to specific questions (e.g., are you more interested in adoption? overall ease of use? functionality?). Finally, pick one or two users for experience sampling. This can be implemented low-tech (good old paper and pencil diary) or higher tech, with a small application installed on the mobile device which alerts them at predetermined or random intervals and asks them questions (again, geared toward the specific questions). Good luck! I'd love to hear what is used and how it works out.

My recommendations span a spectrum from the most qualitative/ethnographic, to the most quantitative/detached. As an additional note - I actually don't think that you'd want to utilize a "conventional user-testing lab" for testing this system anyway, as it's context of use is for mobile scenarios. 1. I firmly believe that you can collect more data in an hour of "shadowing" someone than you can by reading survey responses. I'd recommend recruiting between 8-10 users of the game and meeting up with them at some point during their normal day at a time when they'd be likely to be playing the mobile game. Not knowing the specifics of the game, I'm going to just outline a sample scenario. Let's say a player, Suzanne, likes to play the game while she's on the CalTrain going to and from her workplace. One or two of your researchers can join Suzanne on her train ride and document what she's doing while playing - video-recording and taking notes while Suzanne "thinks aloud." Researchers would want to identify everything from specific design elements that may need to be adjusted for usability, to how playing the games has social impacts on how the users live their life - for instance, how immersive should the game be if a person is playing it while walking down a street? Should it allow for easy pausing of the game as a person needs to switch their attention quickly into their immediate environment, etc.? 2. Another less immersive way of capturing data would be to have the users put on a wearable, automatic camera, such as the SenseCam invented by Microsoft. The SenseCam is worn around the neck and passively, automatically captures thousands of images over a day that can be replayed quickly. Moments when a participant was playing the mobile game can be isolated from other irrelevant shots. You could, for instance, have 8-10 participants wear the SenseCam for 4-5 days. Participants would be invited to come in for an interview at some point during the following week. You could show them a set of instances when they were playing the mobile game and have them recall and walk through what it is they were doing at those moments. One drawback is that the SenseCam, while positioned optimally at a user's midsection, may not record all relevant information on the mobile appliance's screen. Another drawback is that participant's recall of an event is not as robust as actually being there (such as the kind of data you'd be able to collect in 1). 3. Finally, another less immersive and more quantitative way of collecting data would be to push a set of participants surveys to complete 3 times per day (and perhaps, one longer at EOD) about their mobile gaming activity during the interim period. Setting up the right kind of reward structure for quality completion of surveys will be important in this case, as you are relying on the user to accurately and completely capture all of your data about their activity.

2. How can we capture contextual information on the situation users play the game and the features they choose to play? Prepare a questionnaire. Give features as options - to understand what features they used vs. what they haven't and questions against each feature again as to why they used it - because it was easy to use or simplicity. The same is applicable for contextual information again. 3. Having users annotate their thoughts/comments on a form on a daily basis vs. recalling all thoughts at the end of the experience. Simple - good old small diary. Ask them to either jot down the points that immediately strike their mind in whichever form they prefer. Either short sentences or mind maps - whichever is quite convenient for them. They would be more willing to do this and it would be a lot helpful...Trust me on this! 1. How can we collect user feedback in a reliable way without observing our users? What are current field methods used in the mobile world? Keep in touch with the users. Either SMS/call them on an hour to hour basis in the day. Ask them about the questionnaires/diaries - anything different that they have noticed. However, not to much pressure/stress again...

I recommend using Twitter. Various twitter accounts can be setup, each corresponding to an interaction point, condition or state of testing. The portable nature of Twitter, combined with each control timestamped, makes this an effective component of your testing methodology.

One way you could collect user feedback is to send text messages asking them for their input at specified times, either at a point in the game where it is not disruptive, or at a point in the game that is related to the question you want to ask. For example, a text message could ask the user to respond to their dislike or like of a feature shortly after they've used it. This could be basically similar to a thinkaloud protocol or a diary method where the text message can prompt the user. Current field methods include following the person around and giving them tasks to complete, attaching video recording devices or data-recording software that screen captures the actions of the user.

I have two comments/suggestions, one about a specific method, one about overall research perspective and how it guides inquiry. First, you might consider something like the experience sampling method, wherein you randomly prompt participants to elicit their experiences at that moment. This could easily be combined with some of Charlie's previous suggestions about asking participants to leave sharable mp3 voice mails. Granted, you'd probably want some way to align these promptings with when participants are playing the game (though not necessarily), which could make it difficult if you don't want to interrupt them. The specific of how this gets implemented likely have to do with the details of the game mechanics, to which I can't speak here. However, experience sampling might be a way of getting samples of particiants'/players' experiences without being too disruptive. Second, the description above mentions an interest in bridging "the gap between online and offline worlds." Based on my own research, I'm not convinced that people who interact online and offline would describe such a gap in their own experiences. I think we technologists often make this distinction, since there is a clear technical delineation between activities that take place online and those that take place offline, but I'm not convinced that users experience such a strict separation of the two. Rather than having the game attempt to bridge a gap between online and offline, it may be informative to examine how playing the game could blur the distinction; rather than connecting online and offline, perhaps playing the game leads to a decreased in players making a distinction between the two. Rather than online/offline, it may be that a major distinction players make it between teammates/opponents (not sure if this is a cooperative or competitive game) who are physically collocated versus those who are physically disparate. Basically, rather than starting with a priori analytic categories, such as online and offline, it may be more informative to allow such categories to emerge based on analysis users' experiences. Hope that helps, and good luck with your project. ~Eric