Altmetrics are alternative metrics for scholarly output — ways to measure impact,  recognizing that scholarship now happens across media, and impact does not result solely through peer-reviewed journal articles.  Scholarly conversations happen via blogs, via shared software, via re-purposed data, through public “peer review” of shared pre-prints, even on Twitter.  So how can we think we are effectively measuring scholarly impact by counting citations alone?

Jason Priem, a graduate student at UNC in informations science was an electrifying and engaging speaker (although he does, really, talk too fast!).

He started by talking about the first scholarly revolution, where there was effort to standardize the scholarly publication — the “letter” became the basis for the journal article, and very sensibly the format was standardized to clearly present the bits that should be systematically shared (lit review, methodology, findings, etc.).

But now we can use the networked web as the platform for the scholarly record – or, as Priem calls it “web-native science.”  Several things follow from this recognition. First, we can embrace a diversity of scholarly output rather than forcing everyone through a journal article publication tunnel.  And that is already happening – data is being shared, grey literature abounds, significant discussion of results doesn’t wait for publication but happens within networks of people working in the same area. Second, if we see all that as scholarly output, we want to count it, we want to recognize it as worthy output for promotion and tenure.  That’s where altmetrics comes in – how do you measure it, and how do you gather all that disparate data together to tell the story of a scholar’s impact?

A very interesting question is how to ascribe “impact” when discovery is via collective discussion.  One example is “MathOverflow,” a website where people pose problems and everyone who follows that site can suggest solutions or contribute to the analysis.  Just like StackOverflow, a certain community has formed around a common need to have a platform like this.

That knowledge results from community effort is one of the most fascinating aspects of networked culture,  and we have to realize that our image of a lone scholar, solely responsible for her/his own work and able to specifically credit the discrete contributions of others is often inaccurate and in some ways pointless.  Isn’t there significant impact in just asking the right questions in the right places and coordinating input that achieves a solution?  I sometimes speculate what the negative and unintended consequences have been for academia with the promulgation of the lone creator model or the sometimes rigid ways mandated for acknowledging the contributions of others.  I know I say things or think things as a result of all that I have learned in the last decade but I couldn’t begin to untangle what I owe to whom.  Is there a useful way to handle acknowledgement, credit, and authorship that strikes a better balance?   I know that I think about “identifying patterns” as part of learning and knowing, and that I owe some of what I believe about patterns to Josh Lehrer (How We Decide) but what, exactly?  I read it a long time ago and I’ve been thinking ever since….  (I recently watched a documentary about the James Bond books & movies, and there was this endless lawsuit about who “owned” the movie character of James Bond, who was allowed to make the movies. It seemed so sad to me, the premise that there could only be “one James Bond” in the movies, one owner in a winner-takes-all-model.)

Priem uses the phrase “make public” for sharing scholarly output – in direct comparison with the slow and sometimes unfair peer review model of “publishing.”  In Priem’s view “making public” more accurately captures how scholars have impact on others.  You can put something up on a blog and have a far greater impact on the creation of new knowledge than your article — which will only appear 3 years down the road – can hope to have.  In fact, “informal communication” has been recognized as an important aspect of research for at least 60 years.

Ahh, think of every scholar now sharing so much more of what they are doing via the web — tweeting, blogging, grey lit, white papers, group discussions of all kinds.  How will we manage this huge amount of information/emerging knowledge.

Priem showed us briefly how he manages the flow – he organizes his own network via something like Tweetdeck, essentially creating his own “current awareness” journal.

Priem then got into the nuts and bolts of altmetrics, which I’ll just briefly describe.   Once you wade into the open web, how do you measure impact?  He made it look obvious — impact is derived from other peoples’ behaviors.  If you have “made public” your work, then either scholars or members of the general public can do something with it.   They can view it, read it, comment on it, discuss it, share it, save it for future use, cite it, or recommend it.  Clearly some of these activities seem to indicate a deeper engagement than others.  ImpactStory is one response – how to capture the uses made of stuff, weight them as more or less impactful, and create some kind of easily digestible story about the impact of  a scholar’s work, even in some cases being able to benchmark it against other scholarly products (e.g., more “clicked on” than 76% of other stuff in this bucket).  The idea is that a story is more useful than a single numeric indicator of impact, that the story can help reflect the complexity of the ecosystem in which scholarly impact happens.

The non-profit ImpactStory is here:  http://impactstory.org/

–Karrie

David Weinberger

NFAIS 2013 opened with a glorious bit of philosophizing about knowledge in a  keynote from David Weinberger, author of Everything is Miscellaneous and Too Big To Know.

Some people might not care about “what knowledge is,” but surely anyone in or around the field of education gets the importance.  Teensy example – last year I encountered various theories of economic development which my Penn professor at the Graduate School of Education (the very smart Dr. Ghaffar Kucher) related to epistemology, to different paradigms for knowledge.  So I was poised to love Weinberger’s keynote talk, and I did.

Here’s my selective interpretation of Weinberger’s remarks.

The first startling thing is that, Marshall McLuhan-like, what we think knowledge is gets mixed up with the medium by which knowledge is delivered.  When knowledge came in physical containers (books, journals, etc.) you couldn’t physically access it all.  So it was important to condense it, to get value out of it by making it smaller.

We thought of knowledge as being part of a pyramid.  By condensing the raw data into information, people could sift through that and create knowledge.  You don’t have wisdom by just plowing through data, you distill at each layer.  We managed to get to wisdom by “reducing the amount of stuff we have to deal with.”   We filtered stuff out.

Another thing the pyramid shows us about our up-til-now view of knowledge — knowledge has traditionally had the property of being that which is settled.  If we’re still arguing about something, we say we don’t know.  We tend to think of knowledge as something we agree on; in a true state of knowledge there isn’t any more reasonable disagreement.

Theorists who first used the knowledge pyramid explained each layer as built on the other –information is structured data, and knowledge is information-you-can-use.  The model is somewhat limiting in that knowledge, in Western culture going way back, has always been thought of as something more than just a way to get things done.  Knowledge was being truly human, understanding our place in the universe etc.

Other things from the world of knowledge-presented-in-tangible-media.  In the physical world you can only have one organizational scheme – pick one.  So, the taxonomy (where we place an organism in the tree of life, for example) becomes the truth almost.  We forget that’s it’s a representation, a model, and instead we take it for reality.

And that leads us to seeing knowledge as a series of stopping points – we get the answer, boom, move on.

Knowledge in the West  shares the properties of its medium (print).

Enter the Internet.  Our new medium for disseminating knowledge is a NETWORK which gives knowledge the properties of networks.  Networks don’t “end.”  They are infinitely extensible (I think).

Knowledge starts to look more like a network.  (I can see our culture in transition here–many disagreements seem to me based on what counts as knowledge.)

Now compare properties of knowledge from the print world to knowledge in the networked world.  Something sparks a response throughout the network – it’s all relevant and it’s all linked.  Guess what – no one agrees!  (cf the print world, where getting published meant a lot of settled agreement about what you have to say before people would undertake to print your work).  So, if I understand Weinberger right, we begin to have a deeper knowledge because when everyone is saying something (slightly or even dramatically) different, we benefit from a whole lot of available multiple perspectives trained on any particular issue or problem.  And we’re not saying everyone is equal — we’re not saying crazy or unfounded opinions help.  We’re saying these multiple perspectives come from thousands of people with expertise and reasons to weigh in.  We would actually learn less or understand less if all these smart people were saying the same thing.  The value comes from multiple perspectives ad infinitum rather than from filtering for less.

It’s almost paradoxical sounding to hear Weinberger say “Disagreement is how you scale knowledge – it’s how knowledge can get really big.”

He then showed us a diagram with a picture of an ordinary robin in the center, surrounded by labels in all directions to show that when the world at large talks about the robin, it can have many “meanings,” including a robin, a songbird, a symbol of Spring, a disease vector, a work of art, etc etc.  This helps me understand that knowledge is not a finite amount of statements about a robin or one placement of the robin in some ontological scheme.  The knowledge is the infinite number of semantic relationships a robin can be part of.  If the knowledge I’m seeking is the kinds of birds I can see in Medford, Mass. in mid-March, then somewhere there may be a data set with geo-tagged, dated photos of robins. And that’s a pretty mundane example — a robin could be included in climate change knowledge, a study of birds as symbols in literature, or even the “livability” of a city.

If we can cope with the messiness of a network as compared to a pyramid,  knowledge can get really, really big.  Which I take to mean, I could conceivably “know” a lot more because with the right tools I can produce knowledge precisely because I have access to sooooo much information.

Then Weinberger remarked on something that has troubled me ever since I started to work in libraries.  Curation.   In the world of tangible media (where space is an issue) but also in the world of licensed electronic information, you can’t have everything so what do you acquire, manage, keep?  Anybody’s “principles” (e.g., buy what people want) are in conflict with someone else’s (e.g., buy what they’ll need, but don’t know it yet).  It’s so obvious, there is no right answer. (In my public library days, it just depended on who was boss.)  What a relief to have someone talk about how problematic curation is.  You simply cannot predict what will be of interest or need, anymore than you can limit the ways a robin might have meaning.

I do think some libraries got over this, realizing that what their constituents could “access” was just as important as what the library might own.  Still, we spend a lot of time selecting stuff for individual libraries, and as Weinberger pointed out in his earlier book, it’s cheaper to just have everything and filter on the way out.  Selecting, or curating, is a very expensive use of staff time.  Since libraries can’t buy everything (even electronic stuff costs money) they are forced into selecting.  But the illogic is increasingly clear, as libraries struggle with what they should stop doing in order to provide impactful services for users with new needs, and perhaps this will be one of the drivers to radical new business models.

What Weinberger calls filtering on the way out I think I would call discovery — it’s sort of the same thing when you can go to a place where “everything” is, and then drill down to what you want.  You don’t filter by putting only a few things into the bucket; you filter by going into the Bucket of Everything and using discovery tools to bring what you want to the surface.

So, in this new world, we’re no longer going to get value by filtering on the way in, by weeding through and letting only some information become knowledge (by selecting, by only publishing some info, by only providing access to a subset, etc).

Instead, thinking about knowledge by thinking about the properties of networks, Weinberger talked about 3 ways to squeeze more value from information.

#1  Iteration

Iterate forward into knowledge (rather than seeing it as something settled that you obtain.)  His example is the stackoverflow site.  The likelihood that you are the first person to have a problem is minimal to vanishing.  People can help you iterate forward.  Iteration at webscale is an efficient way to produce new knowledge when information is superabundant.

Weinberger alluded to some unfolding cultural effects. We’re still not used to what happens when millions of people can be brought to work together. According to traditional ideas of knowledge production, Wikipedia shouldn’t work, but it does.

 #2.  Platforms

Platforms continually increase value.  I can’t remember what example Weinberger used but the one I have heard is facebook.  Facebook didn’t succeed by putting content out on the web and attracting people to it; it succeeded by being a platform on which people could share content, and by designing the platform so that people’s ordinary interactions could make the content more and more valuable.  If my circle of friends are all recommending a particular book or a data visualization tool, no centralized source is bringing that to my attention, it is the result of many individual behaviors.

Weinberger sees the library as a platform.  The top layer is where the value is created (people using stuff and connecting with other people). As a knowledge network where people discover, build, share.  Everything they are doing in this layer should be feeding back in to your system to keep adding value.  This would be a terrific discussion topic back at the ranch! (It’s the sort of thing that makes me want to talk about why “tagging” in academic libraries never seemed to catch on.)

Again, the top layer is users doing whatever they feel like and adding value by their behaviors.  The middle layer is tools, services that allow users to do “whatever.”  The bottom layer is the data, metadata.

This led to another interesting philosophical point — the only difference between metadata and data in the digital world is functional.  Anything that is data can be metadata for something else. The example is “Melville, Moby Dick, Call me Ishmael.”  Any one of those terms could be the metadata for any of the others, depending on what a user is trying to do.

Similarly to how curation is problematic, Weinberger commented that plans don’t scale.  People who try to anticipate what will happen are going to be wrong.  The pain of being wrong becomes greater as we invest more in our plans.  Seeing the library as a platform allows the development of value without recourse to planning which will be mostly wrong.

 #3.  Linked data

What is happening to knowledge (it’s getting massive and interlinked ) is also happening to data.

Weinberger applauded some experiments with “filtering on the way out” such as Data.gov -  which is full of big, messy stuff.  Just chuck it all in there, and let people develop tools and services around it.

Because data can be meaningfully linked to and used, we don’t need knowledge to be a settled, agreed on thing.  Knowledge is now useful arguments between people…at scale.  The usefulness comes out of linked data.  We can have evidence at our fingertips.  At this point it almost feels better to say that we’re always going to be in the process of knowing rather than to talk of knowledge as if it’s some disembodied thing.  (Although complicated ways of talking like that usually make me impatient….) But it’s really a point to ponder.  What can possibly be seen as a “right answer” when complexity theory demonstrates how mind-blowingly complex problems are, with googobs of interrelated and interacting sub-parts, and we need to think about optimizing in a constantly changing situation rather than getting to one right answer.

So, if we need to have computational ways of making all that data useful to us, we will need semantic search.

Weinberger closed with several points.

These things don’t scale:

• Agreement
• Order
• Control

The web is a good metaphor.  We built this without having control (a project plan, named managers, etc).   There’s no agreement on the web, there’s no real order, no one is in control.

And this Syllogism:

1. Interests are unpredictable
2. Value arises from interest
3. Linked openness enables value to scale

By the way, Weinberger says he works in a basement (at the Law Library at Harvard).  Now that seems too bad.  Let’s hope it’s a nice office down there.

 –Karrie

I haven’t heard evidence for this claim, although one can see how it almost appears to be common wisdom.  But have we got at the truth here?  Untested ideas such as this are the very kinds of potentially wrong hweuristics that ethnographic research is designed to correct.  If you ask students what they want, they will be happy to tell you, but they might provide the wrong solution.  Asking students for their “solutions” is the design charrette approach.  Do you want soft chairs or hard chairs?  Do you want this pink or blue?  Do you want all your courses to have the exact same layout?

I think the more successful approach is participatory service design.

The first step in participatory design is ethnographic research – close observation in a systematic and rigorous way to find out how students interact with courseware.

The second step is rigorously interpreting the data you gather, and applying it to the design problem in order to arrive at an optimal solution.  Interpreting the data and evaluating  potential solutions is done by people whose profession involves service design.

There are many stakeholders in the selection and support of courseware, and one or two platforms on campus will often be the right choice in balancing across constituencies.  And even though that solution may quiet student complaints, it doesn’t address what I’m guessing (and what ethnographic research could determine) is really the source of student complaints — poor design of online learning environments.

I take a lot of classes so I am frequently in the role of student.  But I am not suggesting my perspective leads directly to “the solutions”  for design of online courses.  Being a student just helps me understand why participatory design succeeds. (IT people sometimes call this “eating your own dog food,” meaning that putting yourself in a user’s shoes helps you design better.)

As a student, online courses can annoy me in 3 ways.  There are many, and in my opinion, better solutions to these annoyances than herding everyone into the same courseware corral.   (I’m only looking at a student point of view here; I’ll briefly mention faculty-as-users in a bit.).

My first annoyance is that it is often hard to figure out the navigation of online courses — where has the usual “stuff” been put.  Not everyone takes the same meaning from labels like “syllabus” or “course policies” or “assignments.”  An institution or program could, of course, “design once” and make everyone fit into the same straitjacket — same courseware platform, same course design, same labels, etc.  Then students just need to learn that one pattern. It isn’t hard to see that there will be a downside to this approach, even if it might smooth things over with students in the short run.  Personally, I’d prefer good designs customized to course objectives over sameness.  Some people get course navigation right — it’s like walking into a room where the furniture is arranged nicely versus walking into a room where things are awkwardly placed.  It’s essentially a design question, not a platform or technology problem. So, how costly is it to raise institutional knowledge of course design?  Or, how costly is it to sidestep it?

Another issue for me, wearing my “student hat,” is understanding the affordances of a platform.  Can I plz be told how the tool works in some plain simple way?  If the lecture videos can be slowed down, for example, that’s important to know. Can I run my writing thru a check that will help me improve it prior to turning it in?  Again, research can tell us how best to present explanations so that students take advantage of features and avoid pitfalls.

Annoyance number three involves knowing what I have to do each week.

This is presented in wildly different ways in each course I take – it is not dependent on the platform, it reflects the instructors’ preferences. In the worst cases, weekly lectures are in one place, assignments for submission are described in another, the requirement to post to discussion forums is listed somewhere else.  And as these things inevitably change during a course, instructors sometimes update deadlines in one place but not in all the places where the information is duplicated.  The result is significant administrative overhead for students madly trying to complete weekly requirements.  The problem is not complicated — it’s just this:  the design does not efficiently allow the student to “know” when they are done for the week.  The corollary is inefficiency for the student trying to plan ahead for what they need to do.

I suspect some of these design flaws are actually features for faculty — they are done to minimize the instructor’s administrative overhead either during the course or in migrating the course forward for the next semester.  This is the reason that participatory service design needs to study all users of a system — how do the faculty use courseware?  what about other stakeholders?

My overall point…   Yes, one or two courseware platforms on campus may well be necessary for a host of reasons, but I’m not ready to interpret student complaints as buttressing a “one-size-fits-all” decision about instructional design.   For me, student complaints may lead in a different direction.

One of the IT speakers I heard at ELI a few years back spoke to the problem of balancing standardization and innovation, questioning if short term gains from standardization are outweighed by creating inflexible people in a rapidly evolving climate.  It could be that the really strong institutions of the future will have devoted as much or more time to raising institutional knowledge about the design of learning environments as they do to standardizing on pre-selected platforms.

                                                        –Karrie

I believe the skill required to do that is based on knowledge that has become part of our culture.  Through our ordinary, everyday living and working experiences, we have somehow learned how to organize a productive process that is sequential, linear, modular and cumulative.  And part of our cultural knowledge is knowing when an assembly-line approach is appropriate.  We know almost instinctively how to take advantage of the particular affordances of that kind of process for volume, efficiency and quality control.  We seem to quite naturally understand the difference in affordances between the home-based workshop or skilled artisan, and the collective enterprise that uses the assembly line approach.

What does the future hold regarding our cultural knowledge of productive processes?

Assembly lines are useful in some situations, but we’re increasingly working in a world that needs knowledge workers–handling big, messy, unstructured problems. Knowing how to put together an assembly line is of limited usefulness.

I think the tools of production we need now, and which should seem as easy and natural to us as the organizational tools of the manufacturing era, are tools like agile project management (or even standard project management), change management, establishment and leadership of high performing teams, knowledge management, fostering learning organizations, understanding and nourishing innovation, service design, program assessment, and so on.

We all spend probably quite a lot of time and effort acquiring skills in these areas.  Even so, sometimes it’s hard to get a lot of traction in your workplace for putting your newly acquired skills into common practice.  Legacy “best practices” and “common wisdom” carry cultural weight even when they are no longer yielding the results they once did.

Just as the successful enterprises of the manufacturing era made an asset of their ability to use processes appropriate to their sphere of work, I think organizations today will succeed or stumble based on their ability to make use of the appropriate process, design and management tools.

And that our grandchildren will find it a simple matter to establish and launch high performing teams…                                                                                     Karrie

The meeting focused on undergraduate students.  The first talk was by Susan Gibbons, University Librarian at Yale, on “Enhancing the Undergraduate Library Experience.”  She spoke about the well-known anthropological work done at Rochester, and some about Yale.

Gibbons is a wonderful speaker, zeroing in on the teaching and learning rationale behind innovations.  For example, lots of libraries have adopted cafes or coffee shops within library space “so students don’t have to leave the library.”  Gibbons pointed  to the key teaching & learning reasons for a cafe – designing it to foster the kinds of conversations that enrich and deepen student learning.  She pointed to the coffee shops of the 18^th century as great places for intellectual exploration, and talked about the way comfortable coffeeshops near campus were places where students were intermingling even with faculty and discussing their work.  I’ve certainly seen a lot of library cafes in the last decade.  Some café spaces really foster those conversations, and even attract faculty as well as students.

Designing spaces to foster a community of scholars was a thread throughout her talk.  She told a story about a physics library where group spaces were set up with whiteboards – not only were students gathering there to work on problem sets, but because of the open set up, faculty and grad students would sometimes wander by and volunteer help to students who needed a little extra explanation or coaching.  I’m fascinated by these examples of faculty being in the library…  Again, I would say the key to this innovation is not just “lots of whiteboards” but the activity and interaction that the overall design fostered.

Gibbons briefly pointed to the value of photos as a way to get student answers to questions – something I would like to copy.  Her example was having students take pictures of where in the library they felt lost.  Of course, the stacks were highly featured in these photos, and  in response the library enlisted fellow students–known as stacks ninjas–to help as guides.

She also pointed out how students don’t feel necessarily welcomed or included – and that the library is a welcoming space (or can be) because it belongs to everybody.  It’s neutral.  Her discussion of ethnography also made me think of how students might think they are unwelcome, or be somehow less likely to use, library spaces that seem to be for some particular purpose.  Many libraries I have worked in have lots of “rooms” that seem to be special purpose even though anyone is actually welcome to study in them….

Gibbons highlighted the value of ethnography in breaking down wrong assumptions, such as the idea that students sleep in late.  As has been repeated many times, the initial ethnographic studies showed students were up and about their busy lives early in the morning, often carrying everything they would need for the day, all day long.  And that because they are so busy, they don’t start studying or research work in earnest until late at night much of the time.  Libraries I have worked in have yet to really shift their services substantially toward late evenings….

Gibbons also noted that furniture on wheels not only allowed to students to design optimal work spaces flexibly, but it also allowed the library to quickly reconfigure spaces for different kinds of events.

Gibbons said that in designing collaborative work spaces in the library, it’s helpful to know what spaces on or near campus people already use in that way, and why they like those spaces.  She also remarked that good students can be fairly intentional about the way they “design” their study situations – for example, not bringing a laptop cord, knowing that they then have to buckle down and study before their battery runs low.  Or finding particular spaces in the library that are free of distractions (such as poor internet connections).  Her overall point was, of course, that there are a plurality of study/research space needs even for a single student, and that we need to look at our buildings to see if we’ve provided in a balanced way for these multiple contexts.

A big part of Gibbons’ talk had to do with the way universities have separate administrative units to handle aspects of what seems, to students, like a single messy problem.  Students experience their work as the intermingling of technology, writing, and research – but support is not offered seamlessly for these areas. I think libraries generally know this by now, although it still is a hard problem to solve.

At Rochester, evidently, some librarians were trained as writing tutors, and the writing tutors spent some time at the reference desk.  That seems like a great step forward in reducing the intellectual distance between these services if not the administrative and geographical distance.  Especially in places where a good part of the library’s ability to reach undergrads is through a first year writing course.

Gibbons and the next Yale speaker, Emily Horning, talked compellingly about their Personal Librarian program and how – given the context of Yale and the key features of campus life for freshmen – this was a successful program.  There is a strong identification with one’s college, for example, and that provides a good way for personal librarians to connect with their students.  The actual service uptake is 15-20%.

One of the common themes throughout the day, for me anyway, was the way all these places are leveraging student workers in new ways.  The stacks ninjas were one example (they had a real sense of camaraderie, it was clear from the photos, and had designed their own Tshirt), but several people I spoke to were using students throughout orientation to promote the library.  The student workers were seen almost as advocates in some instances – part of their job was promoting the library.

There was talk throughout the day about the late hours students keep.  One “library opening” after a renovation project happened at midnight and drew a huge crowd of students (the food, if I remember, was gone in 12 minutes …)  Emily Horning also spoke of “late night study breaks” being offered for relaxation during crunch time, realizing that even at that late hour – 11pm or so – students were really just getting started with their studying.

The Yale librarians clearly applied ethnographic research methods and thinking to their own context.  I wonder how well the rest of us would do in answering the question “What are the key defining features of undergraduate life on our campus?” as we seek to develop our services and products?

I was challenged by a colleague to think about what I would be willing to pay for as a student, and of course we are all thinking about the ways the library might contribute to further developments.

Suggestions about online course web sites

Managing Weekly Progress

• a straightforward way (one single place) to know what I must do, how long it will take, easily see that I have completed all steps for each week
• use actual dates rather than the nebulous “week one” and “week two” labels

Improve Overall Course Navigation

• separate course elements into discrete categories so they are easy to find (how things will work or what I call reference material;  video lectures; notes; review materials for tests; etc)
• create a printable course calendar that shows the deadlines  for the entire course.
• enable the course calendar to integrate with all my other Coursera courses if I choose
• create email or other alerts so that I re-visit the calendar when there are changes

Expand ancillary learning materials and create more learning reinforcement

• include more exercises such as glossary exercises, practice questions, etc
• let me see “great examples” of writing about the concepts.  I don’t mean sample assignments, since I think people would tend to just conform and copy what they see, but interesting approaches to either to sub-parts of an assignment or to related assignments given in the past.
• every unit needs a more extensive, non-graded quiz.  This could be started by allowing students to contribute quiz questions, which might double with their “muddiest point” confusions ( see below).
• use “muddiest point” questionnaires to direct students to appropriate ancillary materials (perhaps even just “voted up” content from the discussion forums)
• all math examples or simulations should be separated out for repeated replay at variable speeds, and these should be a priority for development of ancillary materials, esp additional examples of the same concept or different applications of the concept

Improve peer assessments

• let me grade an assignment specifically created for peer assessment training and see how my score stacks up against a standard (this was mentioned as something Coursera is doing but it wasn’t used in my course)
• create more of a template for peer graders to apply the rubric
• allow for a pair conversation as an optional activity.  I agree to view someone’s assignment before we submit, and make suggestions to them.  And they review mine and make suggestions to me.

Raise standards of grading by allowing for revisions

• to prevent grade inflation which I think may be happening (graders are reluctant to give bad marks), allow anyone who has gotten a very poor grade to do revisions, if there are enough volunteers in the class who got good grades to do a couple extra peer assessments.

Use peer assessments to inform teaching

• let peer graders fill out something that indicates what they think students don’t get.  Perhaps a formula sentence like “I think many students may still be unclear about _____” and choose from a list, with the option to fill in a word of their choice.

Expand the Community TA concept

• Allow me to seek a tutor or barter an exchange – I will give feedback on writing quality in exchange for help with a statistical concept. Devise it as a low stakes exchange by limiting it to one week of conversation (in case it doesn’t go well).

Fix the “mosh pit” problem in the discussion forums

• Let me fill in a profile so that I become assigned to be a member of a group where the discussion board is going to be more at my level of discourse
• Let me further refine my ability to engage in ongoing discussion with just the people of my choice – no more mosh pits with 20,000 undifferentiated people.
• Create more structured threads including some short answer Q&A threads that would serve as review and test preparation materials.

Improve tests

• Students want to feel that the test questions got at the most important concepts, and I can’t say I felt that  about my test experiences.  The preponderance of questions toward some topics seemed odd.
• When the right answer is not grammatically responsive to the question, I find it really confusing.  When it doesn’t sound the way a normal person would answer the question, I have a hard time understanding it as the right answer.

Include critical thinking about and evaluation of resources

• Develop at least one unit around the idea that all resources on a topic are not created equal, that we have to evaluate information.  Most faculty have at least one example of poorly presented information or misguided “popularizing” approaches to a topic.
• Use characterization and summarization exercises (these are hard for students to do)
• Clearly label all resources for citation – lectures, guest interviews, etc.  These are learning objects and need a bibliographic status!

MOOCs – What I Would Pay For?

1. A proctored exam in order to get a more meaningful certificate.

2. The ability to download the lectures more easily and quickly to a device, so I can engage with the class in offline situations.

3. A way to find and stay connected with other top performers in the class.

4. A simple way to receive access to the optional materials that I want – like checking off a list of what I want and making one payment, and bingo, it all arrives by some timely date.  (Like a coursepack.)

Library Involvement in MOOCs:   What might we do?

Collaborative assistance in developing the “critical thinking and evaluation of information” unit in the courses.

Assistance with discovery, supply, organization (metadata) and management of copyright-free educational materials.

Assistance with the flow of optional materials to students who paid whatever price was determined.  (The coursepack idea).  [We need to get there first – publishers are actively looking for pay-by-the-drink models and the tens of thousands of potential consumers in MOOCs will be on their radars.]

Assistance with the ancillary learning materials, as they are rated more or less useful by students.  This might involve metadata, search optimization, recommendation systems, categorization not only by topic but by level of sophistication and complexity, presentation interfaces.

Assistance with the development and production of interactive textbooks or learning objects by Penn faculty, using technologies aimed at simplicity and sustainability.

And for on-campus students in blended learning…

Library instruction to help students develop information skills within a discipline can be included in blended courses aimed at on-campus Penn undergraduate and graduate students.  We can and should transform the typical instruction we do in this area.

Faculty and librarians have identified many of the skills gaps that cause students to do poorly in school work based on library research, ranging from developing a good research question to comprehensively exploring relevant literature for a lit review.  Exercises to help students overcome those skill gaps can be designed in an online “computational” environment just as well as exercises for mastering course content.  In other words, we can use computational methods to improve on the one-on-one or classroom-based patterns of traditional librarian teaching.  We can incorporate learning science advances such as the ideas around mastery and the zone of proximal development, for example, which would be next to impossible in the kind of face-to-face teaching we normally do with students.

Freed of the huge operational demands of repeated delivery of library instruction, we can devote our knowledge and abilities to the design of very high quality instruction.

–Karrie Peterson

His examples are from well-known businesses, and  range across:

• HR-type onboarding and training processes
• Knowledge management (WhoWhatWhere at Deloitte – cool!)
• Incentivizing better customer service in various ways, such as encouraging idea-sharing or gamifying the way individuals learn better practices
• Increasing systems thinking about use of professional development funds
• and beyond…

Wow.  Through this course I am realizing how the commercial sector is already integrating gamification into their success in very sophisticated ways.

Over the years, I’ve seen some pretty rudimentary and ham-handed attempts to turn information literacy skills-building lessons into actual games–the gamification class taught me why these didn’t click.  These efforts are not gamification. They were/are infotainment, i.e. an attempt to sugar-coat failed pedagogies.

But back to organizational effectiveness….

Gamification techniques strike me as transformative for internal organizational effectiveness.  Why?

It would be wonderful to have a staff comprised 100% of deeply self-aware people with highly developed motivation to learn from feedback, to solicit criticism of their products, to do the “after action review” for projects that help the organization learn rapidly and broadly.  But that level of self-awareness is pretty difficult to achieve all-day-every-day in real life.  We aren’t all perfect.  And even if many of our colleagues are willing to learn from mistakes, it won’t help if the organization’s culture doesn’t consistently support the kind of openness that fosters learning from mistakes.  Fail early and fail often is a mantra I hear at conferences, but much harder for organizations to foster that culturally.  So, maybe designing our organizations to learn productively using gamification would get us where we want to be a lot faster?

I have thoughts about a “game approach” to help develop project management skills — something that would foster self review by teams and reward feedback loops during projects.  It would be a formative approach, rather than a post mortem that is so dreaded it is often bypassed.

Of course, the examples Werbach draws from business seem somewhat out of reach for academic libraries.  Corporations can fund this kind of organizational effectiveness effort–requiring both design and implementation expertise–with part of their profits.  If you want people to learn before mistakes happen, using gamification to direct staff behavior, you need an informed, articulated program of target behaviors.  Presumably managers could define these targeted behaviors, although I think in actuality many managers would find this harder than expected.

What should we do, how much of our slender operating/innovation funds can be devoted to this kind of organizational re-design?  Among all the risky things libraries need to experiment with in order to adapt, anticipate our users, and offer impactful services, where might gamification for organizational effectiveness fit?

–Karrie Peterson

1) customize learning for individual learners and,
2)  scale it — as a customized experience — to huge numbers of people.

Skillfully using “computational methods” to customize teaching and learning is key for higher education, for libraries.

Because survival, in my opinion, will not come from exclusion, selectivity, making people queue up for your services.  It will come from finding a business model that allows you to be hugely inclusive, and vastly lower the barrier (the “cost” to people) to benefit from your services.

One way to lower “cost” is to have very good customization.  If I am helping someone learn something, the less of their time I waste the better.  Time is wasted whenever teaching and learning is outside the zone of the individual student’s proximal development — it’s too easy or too hard.  Getting a grip on that zone has been sort of an art.  In the library world, we call it the reference interview, and it’s a conversation we often have with users to figure out where they are cognitively, what they do and don’t know about research and research tools.  Traditionally, we’ve done that one-on-one.  Not really scalable (even on a campus with 20,000 students, let alone three times that many.)

Since I became actively involved in learning management systems a few years back, I have been increasingly interested in using online teaching as a way to help students develop information skills.  A lot of what I’ve seen librarians do in blended or online learning is weak – we tend to put links to useful resources into course web sites, or to transfer our one-on-one or group session practices from F2F venues into the online world.  We haven’t used technological affordances to transform our teaching – we’re doing the same things we always have (essentially handouts with links and Q & A) but in a different venue.

Offering googobs of links is not pedagogy.  I would compare this to the early days of MIT’s courseware, where “stuff” was made available, but it wasn’t instructional unless you wanted to put a whole lot of individual effort into it.  A list of potentially useful library resources in a course web site is not, obviously, instructional, and it’s a pretty low level of customization (at the course level rather than the student level).  We can do better, but it matters a lot how we frame the task.

One of the keys for computational solutions is to carefully define the problem you need to solve. An ” art” or something mystical and magical that seems to happen when a librarian helps a student develop research skills doesn’t lend itself to a computational solution.  Computational solutions exist where we can specifically define the types of skills needed, and the knowledge gaps that exist, and design our learning experiences around these articulated problems.  Over time, using learning analytics will enable us to better target our customizations for individuals. But our first task is to work with faculty and students to carefully analyze how students using research tools differ in their typical approaches from what we would consider expert behavior.

We’re not starting from zero – some of this analysis has already been done.  Almost all my conversations with faculty who assign library research, for example, mention the “five perfect articles” syndrome.  Students early in their college careers often begin research by looking for articles that perfectly match their own research question.   They find it challenging to gather relevant information by understanding how each research question has different facets, and how comprehensively understanding the scholarly conversation on your topic involves looking at the literature in these different facets.  This is directly connected to the gap between the way students do keyword searching in a database and the way experts do it.  Expert keyword searching is driven by understanding this facets issue.  Sadly, very many librarian-led search workshops don’t look at the facets problem at all – what they teach is to “think of synonyms and try them.”   That “Just try another one” approach is not helping students learn research as a form of critical thinking.  Imagine if you bought clothes that way — you’d just wander through the clothing department trying on every single pair of pants without ever thinking about why some styles are more flattering than others!

We can do better.  We understand the “five perfect articles” problem and its relationship to the relevant aspects of a research question – now what we need to do is design an engaging and useful online learning environment and put it in the path of students so that they can’t get their research done without experiencing the learning sequence we have designed.

And then go on to the next specifically defined skill/knowledge gap…

–Karrie Peterson

Coursera lets you adopt your level of mastery without any kind of social stigma based on other people judging you.  It’s way more autonomy than a F2F class, where I am pretty sure it will not be good for my reputation to be seen getting a “B” or worse.  In the world of online learning, no one will think less of me for just watching the lectures and absorbing some of the key ideas – I decide what I need.

When I’m going for credentials, I take the tests and try to meet the standards to a high degree – but when I’m just trying to learn something useful, I’d like to decide how far to go.  When learning comes in smaller chunks (6 weeks instead of 16),  it’s a lot easier to schedule into my life.  I believe I could get a masters degree in this way, whereas making the commitment to semester-long courses is hard for me to maintain, given the conflicts with my regular job.

–karrie

I’m a lifelong learner.  I have to be – my profession (librarianship) has been in a constant process of reinventing itself for my entire professional career.

Almost every week I have a panic attack — OMG, I don’t know enough about X. This past week, X was “linked data.”  Often, my panic attack is about job marketability — OMG, if I don’t get a second masters in Y, I won’t have a job in n years.  Y ranges across learning science topics,  educational technology, instructional design and IT management. And depending on my state of mind, n can vary from 2 to 10 years.

So, how do I cope with my panic attacks?

Relentlessly, I sign up for workshops and seminars, take semester-long classes for credit, slog through online credit courses, attend conferences, participate in webinars, organize “lunch and learn” sessions, and read, read, read.  Over the years, my employers have paid thousands of dollars for “professional development” in my field.  I pay too, including hundreds of dollars every year to a company that captures course lectures of premier American college professors and sells them on DVD’s, thought not as credit-bearing courses leading to any credential.

I have enjoyed these various forms of learning, and have found them more or less useful, but the online course packaged the way Coursera has worked it out is winning my allegiance in a big way.

I signed up for way too many courses:  Kevin Werbach’s Gamification, Michael Kearns’ Networked Life, Scott Page’s Model Thinking (which I found late), Christian Terwiesch’s Introduction to Operations Management (not yet started),  and Fatima Wirth’s Fundamentals of Online Education (also not started).

My first taste of Coursera was Gamification, and it was a good choice because it got me thinking a lot about fun and about motivation on many levels.

On one level, I can use the course content in my work — helping students learn information skills.  Information literacy instruction has been kind of siloed as a separate activity for students and I’ve come to believe that instead it should be embedded in all the interfaces of the library, from reference services to discovery tools.

On another level, the Gamification lectures make me think about how I am learning through Coursera, what is working, why it feels fun, what kinds of motivations are involved.

Lectures have gotten kind of a bad rap lately.  I really love lectures, but I love them online much more than in person.  Why?  Because I can stop the lecture at any point when I want to think about what is being said.  You cannot do that with F2F lectures, obviously, and it made me realize that way back in college, I learned NOT to think during lectures because it would cause me to miss out on the next couple points and sometimes even lose the thread of the lecture.  You were, presumably, supposed to furiously take notes and then think about the lecture afterwards.  Frankly, it’s just way more fun to treat an online lecture like half of a conversation with a really intelligent person.  Some lectures, I “talk” almost as much as the professor does!

When I stop the lecture, I might be doing any of these things (none of which I would do in a F2F class):

• Thinking about a point and how it applies to something in my work
• Looking at anything online related to the lecture, to help me really understand and remember
• Vocalizing a point out loud to help myself remember it, to try out how I would say it, especially if I intend to use it
• Looking up the wikipedia entry or some other basic information about a term I have heard but not sure I know what it is (“game theory” for example)
• Emailing a note to a colleague (or to myself) about something mentioned in the lecture
• Reviewing something that went by pretty fast (or because my mind wandered)
• Writing down several questions I might have – and not be missing the next several points

OK, so Professor Werbach is not answering when I stop the lecture to mentally make a “comment” but that is not such a big deal.  It’s not very often in a F2F class that a student gets to ask multiple questions or follow a train of thought with the professor — you share so much air time with the other students that in-class discussion often feels like a fragmented press conference rather than the kind of in-depth conversations you imagine you should be having in academia. And anyway, I know that when I amplify the lecture in my own mind–creating my own examples to apply what I am learning–it’s not going to be that interesting to just anyone.  But being able to stop the lecture means I can think through the ideas that are sparked in my mind and guess what– I remember the lecture without having to take notes.  It becomes part of what I “know” without memorizing or cramming. That’s how I learn in real life, after all.

How obvious – that anything is more interesting if you can think about it! I have autonomy! I’m not denying that collective learning can also be powerful, and that I (currently) have less of that by taking my online courses as a solitary student for the most part.  But I have sat through plenty of class discussions that did nothing for my learning and were not even approaching what I would call fun.  Group discussion is not automatically collective learning any more than “hands-on time” is automatically active learning.  If the tradeoff is losing the classroom group discussions (which were often  a waste of time) in order to get the benefits of online learning, my choice is clear.

Which brings me to make a point about the discussion forums online.  I looked at the discussion in Gamification once or maybe twice, and posted only one comment.  Being in a mosh pit with tens of thousands of other students in a completely undifferentiated way is not appealing to me at all.  Fortunately, it should not be hard to use technology, personally drafted profiles, and other basic preferences and structures to enable the formation of smaller groups that will benefit more from open discussion.  Using optional tools, I could have potentially found other people I know in the course — completely by chance in F2F conversation I found out that a colleague in my own department was taking the course.  What would be perhaps even more interesting (and this was a feature of Sakai OAE) would be to connect with other like-minded individuals who share the same “problem-space” that I do but who are taking different classes!  You always meet at least some interesting people in a face-to-face class, and Coursera would be even more useful to me if it could help me build networks with real utility for my professional work.

The other aspects of the course design I like are the way the lectures are broken up in smaller segments.  The technology doesn’t work very well for replaying segments, so it’s just easier sometimes to replay short lectures from the beginning.   I suspect in the future, we’ll all be able to click on a powerpoint slide and get right “into” the lecture at that point.  That kind of repeatability in small bites will be great.

I also like it when I can see even before I click the link how long a lecture is.  It seems like a small thing, but it’s a highly relevant planning tool. Online learning is a very time-conscious, shooting-for-maximum-efficiency kind of enterprise for me, so anything that helps me on that road is a valuable feature.

I like the very easy “mid-lecture” quiz questions, and I think a lot more could be done with those.  I miss not having more online reinforcement of what I’m learning — perhaps more short non-credit quizzes, or a worksheet for me to fill out, or a glossary exercise of some kind, or even just more “what we just learned” review lectures.  I’m not really going to take notes (how laborious, and a boring way to get reinforcement).  I don’t think I should have to do that in order to robustly learn the material, but I need some other fun and fast form of reinforcement and consolidation.  What I’ve tended to do to create reinforcement is take the homework quiz right after watching all the lectures–the quiz becomes the reinforcement.

One note – in Page’s course, the mid-lecture online quizzes are a lot harder than in Werbach’s Gamification course.  I appreciate the “pat on the back” feeling I get from succeeding at the easy mid-lecture quiz questions (don’t take them away!).  At the same time,  it would be nice to have some intermediate step between the easy mid-lecture quiz questions and the actual homework quizzes that count.  I don’t have a good way to test that I know the material before I get into the summative assessment tool.

I note also that I really like seeing the professor talking on my screen, and I can’t say why.  The courses where I see the professor seem a lot more engaging than just hearing a voice and seeing slides.

Overall, on most of the courses, the main thing I don’t like is the presentation of what you have to do each week to stay within course deadlines.  A syllabus seems like a dumb thing to me — it’s become this kind of catchall for rules and obligations, explanation of learning goals, course overview and schedule.  Most of that is reference material I will read once and then keep in the background somewhere – a syllabus is a yucky way to see the schedule.  I want a simple place to know what I have to do each week (watch these lectures, take this quiz, etc) and I don’t want it expressed as “Week 1″ or “Week 4.”  I don’t keep track of my progress in any course by week number, it’s a calendar label that isn’t used anywhere else in my life.  I know that’s easier for the teacher, but piecing together which lectures go with which quizzes and which deadlines correspond with which weeks is unnecessary cognitive load from the student point of view, and gets really difficult when you have more than one class with different starting weeks.  I can think of many navigational ways to make this better in courses, and surely several of those ways would not place too much burden on the professor trying to recreate the class in the future.

Parenthetically, I’ve heard it said that students don’t want a lot of different interfaces, they just want to use one courseware system on campus.  I don’t agree with that.  Students use a huge number of interfaces everyday on their phone apps alone.  I think what they (we) don’t like is how confusing it is to figure out what you are supposed to do.  One way to handle that is to standardize all courses so they look exactly the same – students figure it out for one course, and then follow that pattern for all other courses.  A better way, in my opinion,  is to use good design for course navigation and schedules, and allow students to have an integrated calendar so they can see all their deadlines at once.

Some of the courses I signed up for, Networked Life and Model Thinking, have some (sigh) math.  In almost every instance, the professor goes too fast for me — I’m sure they would feel awkward and stupid to talk through a math example as slowly as would enable me to grasp it the first time, but I could really suggest that they re-film these segments and offer us the choice of watching some bits at a slower pace.

Sadly, I’m likely to drop out of these courses as the math becomes more important to the lectures because in their current instantiation, there aren’t enough examples and ancillary materials to help me efficiently grasp the mathematical concepts.  I can hear the professors thinking “well, you have to expect to do some work to master these math concepts” but I’m convinced that I could master these concepts without (inefficiently) struggling if the pedagogy was more robust.    What I want is a lot of ancillary and optional video clips explaining the concepts using lots of different examples, so I can fly through all of them until I find the one that clicks with me, or use the variations to enrich my understanding.  I am not going to sit with pencil and paper, replaying the same lecture segment over and over, and struggling to understand the concept that the professor handles so expertly and elegantly.  I know from experience that struggling on my own with problem sets is a) unnecessarily time consuming and that b) I could potentially simply fail to ever grasp the concept fully.  I don’t believe that there has to be pain and suffering for learning to happen, any more than  I think it’s better to have lost ten pounds eating only food that you hate.

Why am I in favor of ancillary materials to watch — wouldn’t that also be time consuming? Yes, but I can be doing my laundry or washing dishes at the same time, whereas in my pencil and paper example, the cognitive load is so heavy I can’t do anything else.

Does that shock you?

Learning has been conflated with suffering so long that I think people get uncomfortable with the idea that you can learn instantly from just the right examples.  We use simulations for that reason, and they are way less time consuming in making their point than pen and pencil problem sets.  So, you see, I believe that fast learning is every bit as character-building as slow, inefficient learning! Both Networked Life and Model Thinking make excellent use of simulations.

So, how could all these supplemental video clips with many different examples be created?  One possible way is to have students make them, and the best ones could become featured though a combination of student peer review and voting.  I am in favor of student-created content because I think I am more likely to get an explanation that clicks with me from someone who has only recently risen above my level of confusion — experts sometimes are just too advanced to see the best ways to get things across to novices. But really, there are a million different ways to handle this kind of “teaching for understanding”:

• an FAQ based on common student misunderstandings
• a guided tutorial
• excerpts from other written material
• simulations
• even watching a very slow step-by-step repetition of the example used.

Maybe some students would “pass” an exam question on a concept, and become eligible to host a chat session in which they tried to help other students.  Seeing the course more as a learning community with many potential teachers would allow for leveraging the varying abilities of students in the course–you just need some mechanism for identifying people for various roles.  I might, for example, be willing to help a student with proper writing if they could help me with math.

I like the peer grading that we do in Professor Werbach’s class for the written assignments.  It was useful to see how other people approached the question posed and reminded me of concepts I overlooked in applying the lecture to the assignment.  The feedback I got on my own assignments were kind of the same as what you get on the job — some criticism you take to heart, and other criticism seems not really relevant or sometimes even just wrong.

I think as these Coursera courses become more mature, a lot more can be done with peer interaction, including some group projects.  Matching people in groups algorithmically (based on self-descriptions, perhaps) when you are drawing from a  pool of thousands seems likely to me to work better than getting assigned to a group in a classroom with 30 students.

I haven’t really liked the writing assignments themselves, though, in Gamification.  They are not “in my field” so to speak, where I am eager to apply the concepts I am learning.  I’m not interested in gamifying to sell more breakfast pastries, but I am interested in gamifying to help students learn information skills.  With tens of thousands of students, it seems that in the future it would be possible to offer some choices in the assignment so that students would peer grade on written assignments more aligned with their native interests. [Note: The final writing assignment about e-books was really relevant to my area of interest.]

I adore the short time frame of these classes, because sometimes when I want to know about a new subject, I don’t want a whole semester on it – I just want to understand the area better than I do.  So, I much prefer the amount of content that can be delivered in a 4-8 week course, with chances to further my knowledge in other short bursts.  It’s the way I read books.  Sometimes I’ll read 2 or 3 books on a topic, but very often, I will read one book on a topic and move on to something else, circling back to the first topic later.

Another reason for shorter classes has to do with convenience.  I can fairly accurately predict whether or not I have the “space” for something additional in my life 6 to 8 weeks in advance.  Longer stretches, and it gets iffy.

That said, the pace is a bit relentless, and I’m not sure why these courses have such tight time frames.  Having caught a cold, I really struggled to keep up with even one course, let alone more than one.  I have heard a lot about students not staying motivated in online courses and that you have to structure in a lot of discipline, and perhaps the thinking is that if you give people enough scheduling leeway, they’ll just give up on the course.  I wonder about the option of letting me choose at the outset of a course if I want to cover the material in 8 weeks or 10, and letting a computer compute my deadlines and match me up with other students on my same schedule.

And finally … I have a funny ambivalence about getting the certificate.  I am determined to get at least one.  I feel that I could put this on my resume, but that I could not put a course on my resume if I didn’t get the certificate.  For example, if I stick with Page’s course on Model Thinking but take all the quizzes so late I don’t qualify for the certificate (which is likely to happen), I feel I can’t claim to have taken the course because it’s not quite verifiable.

A completely gossipy aside – I first came across Scott Page via the Teaching Company and his lecture series on complexity theory.  And last year, I met in person a professor whose lectures on ancient Greece had been a Teaching Company favorite of mine.  These guys are my celebrities — I couldn’t probably identify any movie star of recent date, but I’m kind of jazzed to think of these very smart people as my, well, virtual friends.

–Karrie Peterson