Exploring Alternative Visions in Assessing Informal Learning Environments

A Few Moments with Vera Michalchik

Vera Michalchik is Director of Research on Informal Learning Environments in SRI International’s Center for Technology in Learning. For more than two decades, she has been researching the relationships and differences between in school and out of school learning. Michalchik is one of the key point people working on the Connected Learning Research Network longitudinal survey, and she is highly involved in the next phase of digital media and learning work, creating an assessment clearinghouse that will bridge the gap between the research work and design work of the initiative. In the past three years, she has co-authored numerous reports and publications on assessment and evaluation including Naturalizing Assessment and Documenting and Assessing Learning in Informal and Media-Rich Environments. We spent a few moments with Michalchik picturing what assessment practices ought to look like in today’s networked world where learning is now less predictable and more diverse than ever.

How has the purpose of assessment changed over the last decade as learning starts to occur anytime, anywhere, without any control by learning professionals?

There are two countervailing forces out there right now. One is a real push to recognize the importance of informal learning environments. We have seen a sea change in recognizing that learning is “lifelong, life wide, and also life deep” – as it’s been coined. It’s not just lifelong and across spaces in one’s life, home, community etc., but there is also a depth of learning that occurs when you look beyond simply what happens in school and look at what people value most. As soon as you get into this new paradigm of thinking about when and where learning occurs and when and where learning is important, suddenly the old tools for assessment, which are really built around some fairly antiquated views and notions of learning, become rather at odds with and, in some ways, antithetical to learning.

Assessment, as it has been traditionally defined and practiced, has been about sequestering students apart from a situation of learning and resources that would allow them to apply those resources to a problem. The intent has been to isolate and identify stuff in the head that stands apart from the world, which is out of touch with the ways people are using technological resources to support their learning. 

A recognition of learning as lifelong, life wide, and life deep, stands in contrast to assessment techniques and tools that belong to a behaviorist paradigm of knowledge accrual. There is a movement among some who are involved in the informal or after-school space who feel we need to standardize ways to measure learning outcomes, while others fear this is another version of school encroaching on the out-of-school or after-school space. We are at risk of inverting the situation where we celebrate and recognize that there are so many possibilities for creating engaging, dynamic, and meaningful learning environments, but with pressure from funders and desire to prove there is learning happening, we end up with a push to standardize metrics and potentially create assessments that may not be that different from what we have seen in school. We risk becoming out of touch with the ethos of “not school” and might potentially undermine the diversity and the strengths of informal learning environments. 

How do you respond to these pressures to apply standardized measures to informal learning environments?

We need to be careful in our pursuit of coming up with common metrics and trying to apply those widely. One concern is about the diversity of learning experiences in learning environments. I’m paraphrasing Richard Feynman who said that the more that we have a monoculture of learning, the less chance we have of producing creative, innovative, capable thinkers. We really want diverse learning environments, and assessment is always the tail that wags the dog. People are beholden to systems of accountability, and what knowledge is valued and how that knowledge is valued really shows up in an assessment system. Besides reducing the diversity of learning environments by having common metrics, we short-change a natural process. This is what we mention in the Naturalizing Assessment article. People are darn good at assessing who knows what in a very intuitive way, and there is potentially a really good analysis to be had historically and otherwise, that knowing who knows how to do what, being able to determine this pretty quickly, and then being able to optimize human and other resources to accomplish a task at hand is an essential function of social interaction.

Can you give an example of an informal learning environment that effectively utilizes these new types of assessment? 

With some of these high-touch programs where there is tight mentorship and a lot of scaffolding of participants’ activities, there is a really nice continual monitoring of where kids are in the program. I think it would be possible to systematize a lot of the insights and intuitions that facilitators or mentors have in those situations so that you could readily say at what point the kid is engaged and “getting it,” this is usually what we see, and if we don’t see this, they are not getting enough support. Anything that is formative and closely tied to the actual experience and grounded in the learning experience is quite important and valuable, and there are programs that do that well such as the Computer Clubhouse.

And several teams have looked at learning outcomes in Computer Clubhouse activities – in relation to face-to-face activities, life outcomes, and the products members create. In general, there are more and more systematic studies of behavior and performance that are minimally disruptive to the informal activity but also allow for documentation of what kinds of participation occur and capacities get put to use in a learning environment, and also what types of choices follow from experience. Ethnography can be used to demonstrate how a type of learning environment or system works, what kinds of outcomes follow from participation. Video documentation, data mining methods of video archives, embedded assessment in learning games are some of the ways of looking for informal learning outcomes that can be more wide-reaching and scalable. 

Do you see digital badging as a valid type of assessment?

I have varied thoughts on badging. I am a little bit in the “I want to see” camp. The one thing you never want to do, and this is a critical finding from social psychology, is offer an incentive in situations where people might be inherently motivated to do something, because as soon as you offer an incentive, you suggest an incentive is needed. I worry about badges becoming an incentive where there is a deeper and more authentic, interest-based motivation or reason to participate available. This is my first concern. 

My second concern is that badging can be gamed like any assessment system, and this shouldn’t be about giving people who are going to game the system more chances do so. Badging at its heart should be about giving people a way to show the world something valuable they have done – this is the core piece – so they can demonstrate they have gotten something out of participating in an informal learning environment. This is essential, as is providing post hoc social recognition or appreciation. I totally understand the intent of creating an alternative and more nimble, specialized credentialing form, but how this actually plays out will be really powerful and helpful in certain environments and less so in others. We really need to wait to see how it plays out.

I was a Girl Scout leader for a long time. It’s such a closed system, with badges mattering for the pleasure and aesthetics of scouting but otherwise in relatively inconsequential ways – unless girls complete Silver or Gold Awards, which the larger community and college admissions officers pay attention to. I could see both the good and the bad in the microcosm of the badge system. It could be playful and fun, but it can also take away from what matters in context, and as far as reputation and certification go, I’d like to really see, empirically, whether badges do more for people than would learning how to write a resume or learning how to articulate what it is they have learned in a non-school environment. But that remains to be seen. 

What are your hopes for the future of assessment?

We need to start lobbying as parents and families of children in school. That’s really an important pressure point. I talk to parents so often, and when they mention an activity their child is doing outside of school, they tell me that their child is so happy and he/she is learning this or that, but then the conversation switches to school mode, and they start discussing how the child has to get this or that grade. They are of two minds. We have two competing paradigms right now – one is tenacious and longstanding, and the other is making inroads. A lot of this battle is a matter of rhetoric, social marketing, and being persistent in providing an alternative vision. 

As soon as we have this alternative vision that learning is not about knowledge accumulation and test performance, but about participating in activities that are well designed or that naturally provide a lot of opportunity to become better at something, we will start to shift the balance and start to create opportunities to value and honor things that haven’t been honored or valued in school very much. 

There are, of course, intransigent parts of the system, and there are policy issues that are enormous and complicated, but hopefully we can start to value the possibility of making school environments more like out-of-school environments in terms of the kinds of activities and the kinds of informal, interactional assessments that take place in those settings. In those settings, it’s much more about kids trying, maybe failing, and maybe succeeding, all the while engaging with the materials and each other and doing so in ways that show they are attending to the resources and the possibility for building skills in that environment that help them solve a problem, accomplish a goal, or succeed at a game. 

Do you have any story in particular that illustrates we’re beginning to move in a promising direction?

A science center in Texas had worked briefly with teachers and taught them how to make Scribble Bots. The teachers took the Scribble Bots back to their classrooms, and some time later, they administered their standardized tests. The test scores were basically the same as before they had used the Scribble Bots, except for one little peak. They dove in deeper to find out what it was, and it was a science question on circuitry. The students had all scored ridiculously high on the question. Teachers don’t want to teach for the test, but it’s true you learn the basics of circuitry when you use these types of devices. You’re never ever going to forget the basics of circuitry. In the world we’re living in right now there’s really no value in accumulating lots of knowledge, but instead there’s value in having a foundational sensibility and appreciation of a domain of activity and knowledge enough where you can leverage and capitalize on that foundation so you can do more, choose to do more, in the future. 

When kids hang out in a makerspace and use a Makey Makey or some other device, if nothing else, there is a physical and embodied experience that changes their relationship to an important aspect of scientific knowledge, which is the circuit. There are so many stories out there like this, stories that affirm learning is happening when people are creating technical systems, products, and artifacts, and this kind of learning can’t be ignored. We need to document and share it in ways that are true to the nature of the learning and are also convincing to the world at large.

Photo credit: Exploratorium