NFAIS 2013 – Altmetrics, ImpactStory, Jason Priem

Slowly I’m summarizing a few of the great presentations at NFAIS 2013.

Altmetrics are alternative metrics for scholarly output — ways to measure impact,  recognizing that scholarship now happens across media, and impact does not result solely through peer-reviewed journal articles.  Scholarly conversations happen via blogs, via shared software, via re-purposed data, through public “peer review” of shared pre-prints, even on Twitter.  So how can we think we are effectively measuring scholarly impact by counting citations alone?

Jason Priem, a graduate student at UNC in informations science was an electrifying and engaging speaker (although he does, really, talk too fast!).

He started by talking about the first scholarly revolution, where there was effort to standardize the scholarly publication — the “letter” became the basis for the journal article, and very sensibly the format was standardized to clearly present the bits that should be systematically shared (lit review, methodology, findings, etc.).

But now we can use the networked web as the platform for the scholarly record – or, as Priem calls it “web-native science.”  Several things follow from this recognition. First, we can embrace a diversity of scholarly output rather than forcing everyone through a journal article publication tunnel.  And that is already happening – data is being shared, grey literature abounds, significant discussion of results doesn’t wait for publication but happens within networks of people working in the same area. Second, if we see all that as scholarly output, we want to count it, we want to recognize it as worthy output for promotion and tenure.  That’s where altmetrics comes in – how do you measure it, and how do you gather all that disparate data together to tell the story of a scholar’s impact?

A very interesting question is how to ascribe “impact” when discovery is via collective discussion.  One example is “MathOverflow,” a website where people pose problems and everyone who follows that site can suggest solutions or contribute to the analysis.  Just like StackOverflow, a certain community has formed around a common need to have a platform like this.

That knowledge results from community effort is one of the most fascinating aspects of networked culture,  and we have to realize that our image of a lone scholar, solely responsible for her/his own work and able to specifically credit the discrete contributions of others is often inaccurate and in some ways pointless.  Isn’t there significant impact in just asking the right questions in the right places and coordinating input that achieves a solution?  I sometimes speculate what the negative and unintended consequences have been for academia with the promulgation of the lone creator model or the sometimes rigid ways mandated for acknowledging the contributions of others.  I know I say things or think things as a result of all that I have learned in the last decade but I couldn’t begin to untangle what I owe to whom.  Is there a useful way to handle acknowledgement, credit, and authorship that strikes a better balance?   I know that I think about “identifying patterns” as part of learning and knowing, and that I owe some of what I believe about patterns to Josh Lehrer (How We Decide) but what, exactly?  I read it a long time ago and I’ve been thinking ever since….  (I recently watched a documentary about the James Bond books & movies, and there was this endless lawsuit about who “owned” the movie character of James Bond, who was allowed to make the movies. It seemed so sad to me, the premise that there could only be “one James Bond” in the movies, one owner in a winner-takes-all-model.)

Priem uses the phrase “make public” for sharing scholarly output – in direct comparison with the slow and sometimes unfair peer review model of “publishing.”  In Priem’s view “making public” more accurately captures how scholars have impact on others.  You can put something up on a blog and have a far greater impact on the creation of new knowledge than your article — which will only appear 3 years down the road – can hope to have.  In fact, “informal communication” has been recognized as an important aspect of research for at least 60 years.

Ahh, think of every scholar now sharing so much more of what they are doing via the web — tweeting, blogging, grey lit, white papers, group discussions of all kinds.  How will we manage this huge amount of information/emerging knowledge.

Priem showed us briefly how he manages the flow – he organizes his own network via something like Tweetdeck, essentially creating his own “current awareness” journal.

Priem then got into the nuts and bolts of altmetrics, which I’ll just briefly describe.   Once you wade into the open web, how do you measure impact?  He made it look obvious — impact is derived from other peoples’ behaviors.  If you have “made public” your work, then either scholars or members of the general public can do something with it.   They can view it, read it, comment on it, discuss it, share it, save it for future use, cite it, or recommend it.  Clearly some of these activities seem to indicate a deeper engagement than others.  ImpactStory is one response – how to capture the uses made of stuff, weight them as more or less impactful, and create some kind of easily digestible story about the impact of  a scholar’s work, even in some cases being able to benchmark it against other scholarly products (e.g., more “clicked on” than 76% of other stuff in this bucket).  The idea is that a story is more useful than a single numeric indicator of impact, that the story can help reflect the complexity of the ecosystem in which scholarly impact happens.

The non-profit ImpactStory is here:  http://impactstory.org/

–Karrie

Knowledge as a Network; Libraries as platforms

David Weinberger

NFAIS 2013 opened with a glorious bit of philosophizing about knowledge in a  keynote from David Weinberger, author of Everything is Miscellaneous and Too Big To Know.

Some people might not care about “what knowledge is,” but surely anyone in or around the field of education gets the importance.  Teensy example – last year I encountered various theories of economic development which my Penn professor at the Graduate School of Education (the very smart Dr. Ghaffar Kucher) related to epistemology, to different paradigms for knowledge.  So I was poised to love Weinberger’s keynote talk, and I did.

Here’s my selective interpretation of Weinberger’s remarks.

The first startling thing is that, Marshall McLuhan-like, what we think knowledge is gets mixed up with the medium by which knowledge is delivered.  When knowledge came in physical containers (books, journals, etc.) you couldn’t physically access it all.  So it was important to condense it, to get value out of it by making it smaller.

We thought of knowledge as being part of a pyramid.  By condensing the raw data into information, people could sift through that and create knowledge.  You don’t have wisdom by just plowing through data, you distill at each layer.  We managed to get to wisdom by “reducing the amount of stuff we have to deal with.”   We filtered stuff out.

Another thing the pyramid shows us about our up-til-now view of knowledge — knowledge has traditionally had the property of being that which is settled.  If we’re still arguing about something, we say we don’t know.  We tend to think of knowledge as something we agree on; in a true state of knowledge there isn’t any more reasonable disagreement.

Theorists who first used the knowledge pyramid explained each layer as built on the other –information is structured data, and knowledge is information-you-can-use.  The model is somewhat limiting in that knowledge, in Western culture going way back, has always been thought of as something more than just a way to get things done.  Knowledge was being truly human, understanding our place in the universe etc.

Other things from the world of knowledge-presented-in-tangible-media.  In the physical world you can only have one organizational scheme – pick one.  So, the taxonomy (where we place an organism in the tree of life, for example) becomes the truth almost.  We forget that’s it’s a representation, a model, and instead we take it for reality.

And that leads us to seeing knowledge as a series of stopping points – we get the answer, boom, move on.

Knowledge in the West  shares the properties of its medium (print).

Enter the Internet.  Our new medium for disseminating knowledge is a NETWORK which gives knowledge the properties of networks.  Networks don’t “end.”  They are infinitely extensible (I think).

Knowledge starts to look more like a network.  (I can see our culture in transition here–many disagreements seem to me based on what counts as knowledge.)

Now compare properties of knowledge from the print world to knowledge in the networked world.  Something sparks a response throughout the network – it’s all relevant and it’s all linked.  Guess what – no one agrees!  (cf the print world, where getting published meant a lot of settled agreement about what you have to say before people would undertake to print your work).  So, if I understand Weinberger right, we begin to have a deeper knowledge because when everyone is saying something (slightly or even dramatically) different, we benefit from a whole lot of available multiple perspectives trained on any particular issue or problem.  And we’re not saying everyone is equal — we’re not saying crazy or unfounded opinions help.  We’re saying these multiple perspectives come from thousands of people with expertise and reasons to weigh in.  We would actually learn less or understand less if all these smart people were saying the same thing.  The value comes from multiple perspectives ad infinitum rather than from filtering for less.

It’s almost paradoxical sounding to hear Weinberger say “Disagreement is how you scale knowledge – it’s how knowledge can get really big.”

He then showed us a diagram with a picture of an ordinary robin in the center, surrounded by labels in all directions to show that when the world at large talks about the robin, it can have many “meanings,” including a robin, a songbird, a symbol of Spring, a disease vector, a work of art, etc etc.  This helps me understand that knowledge is not a finite amount of statements about a robin or one placement of the robin in some ontological scheme.  The knowledge is the infinite number of semantic relationships a robin can be part of.  If the knowledge I’m seeking is the kinds of birds I can see in Medford, Mass. in mid-March, then somewhere there may be a data set with geo-tagged, dated photos of robins. And that’s a pretty mundane example — a robin could be included in climate change knowledge, a study of birds as symbols in literature, or even the “livability” of a city.

If we can cope with the messiness of a network as compared to a pyramid,  knowledge can get really, really big.  Which I take to mean, I could conceivably “know” a lot more because with the right tools I can produce knowledge precisely because I have access to sooooo much information.

Then Weinberger remarked on something that has troubled me ever since I started to work in libraries.  Curation.   In the world of tangible media (where space is an issue) but also in the world of licensed electronic information, you can’t have everything so what do you acquire, manage, keep?  Anybody’s “principles” (e.g., buy what people want) are in conflict with someone else’s (e.g., buy what they’ll need, but don’t know it yet).  It’s so obvious, there is no right answer. (In my public library days, it just depended on who was boss.)  What a relief to have someone talk about how problematic curation is.  You simply cannot predict what will be of interest or need, anymore than you can limit the ways a robin might have meaning.

I do think some libraries got over this, realizing that what their constituents could “access” was just as important as what the library might own.  Still, we spend a lot of time selecting stuff for individual libraries, and as Weinberger pointed out in his earlier book, it’s cheaper to just have everything and filter on the way out.  Selecting, or curating, is a very expensive use of staff time.  Since libraries can’t buy everything (even electronic stuff costs money) they are forced into selecting.  But the illogic is increasingly clear, as libraries struggle with what they should stop doing in order to provide impactful services for users with new needs, and perhaps this will be one of the drivers to radical new business models.

What Weinberger calls filtering on the way out I think I would call discovery — it’s sort of the same thing when you can go to a place where “everything” is, and then drill down to what you want.  You don’t filter by putting only a few things into the bucket; you filter by going into the Bucket of Everything and using discovery tools to bring what you want to the surface.

So, in this new world, we’re no longer going to get value by filtering on the way in, by weeding through and letting only some information become knowledge (by selecting, by only publishing some info, by only providing access to a subset, etc).

Instead, thinking about knowledge by thinking about the properties of networks, Weinberger talked about 3 ways to squeeze more value from information.

#1  Iteration

Iterate forward into knowledge (rather than seeing it as something settled that you obtain.)  His example is the stackoverflow site.  The likelihood that you are the first person to have a problem is minimal to vanishing.  People can help you iterate forward.  Iteration at webscale is an efficient way to produce new knowledge when information is superabundant.

Weinberger alluded to some unfolding cultural effects. We’re still not used to what happens when millions of people can be brought to work together. According to traditional ideas of knowledge production, Wikipedia shouldn’t work, but it does.

 #2.  Platforms

Platforms continually increase value.  I can’t remember what example Weinberger used but the one I have heard is facebook.  Facebook didn’t succeed by putting content out on the web and attracting people to it; it succeeded by being a platform on which people could share content, and by designing the platform so that people’s ordinary interactions could make the content more and more valuable.  If my circle of friends are all recommending a particular book or a data visualization tool, no centralized source is bringing that to my attention, it is the result of many individual behaviors.

Weinberger sees the library as a platform.  The top layer is where the value is created (people using stuff and connecting with other people). As a knowledge network where people discover, build, share.  Everything they are doing in this layer should be feeding back in to your system to keep adding value.  This would be a terrific discussion topic back at the ranch! (It’s the sort of thing that makes me want to talk about why “tagging” in academic libraries never seemed to catch on.)

Again, the top layer is users doing whatever they feel like and adding value by their behaviors.  The middle layer is tools, services that allow users to do “whatever.”  The bottom layer is the data, metadata.

This led to another interesting philosophical point — the only difference between metadata and data in the digital world is functional.  Anything that is data can be metadata for something else. The example is “Melville, Moby Dick, Call me Ishmael.”  Any one of those terms could be the metadata for any of the others, depending on what a user is trying to do.

Similarly to how curation is problematic, Weinberger commented that plans don’t scale.  People who try to anticipate what will happen are going to be wrong.  The pain of being wrong becomes greater as we invest more in our plans.  Seeing the library as a platform allows the development of value without recourse to planning which will be mostly wrong.

 #3.  Linked data

What is happening to knowledge (it’s getting massive and interlinked ) is also happening to data.

Weinberger applauded some experiments with “filtering on the way out” such as Data.gov -  which is full of big, messy stuff.  Just chuck it all in there, and let people develop tools and services around it.

Because data can be meaningfully linked to and used, we don’t need knowledge to be a settled, agreed on thing.  Knowledge is now useful arguments between people…at scale.  The usefulness comes out of linked data.  We can have evidence at our fingertips.  At this point it almost feels better to say that we’re always going to be in the process of knowing rather than to talk of knowledge as if it’s some disembodied thing.  (Although complicated ways of talking like that usually make me impatient….) But it’s really a point to ponder.  What can possibly be seen as a “right answer” when complexity theory demonstrates how mind-blowingly complex problems are, with googobs of interrelated and interacting sub-parts, and we need to think about optimizing in a constantly changing situation rather than getting to one right answer.

So, if we need to have computational ways of making all that data useful to us, we will need semantic search.

Weinberger closed with several points.

These things don’t scale:

• Agreement
• Order
• Control

The web is a good metaphor.  We built this without having control (a project plan, named managers, etc).   There’s no agreement on the web, there’s no real order, no one is in control.

And this Syllogism:

1. Interests are unpredictable
2. Value arises from interest
3. Linked openness enables value to scale

By the way, Weinberger says he works in a basement (at the Law Library at Harvard).  Now that seems too bad.  Let’s hope it’s a nice office down there.

 –Karrie

Courseware, students, participatory design . . .

I sometimes hear educators say that students want only one courseware to contend with, that students don’t want to deal with multiple platforms for the courses they take each semester.

I haven’t heard evidence for this claim, although one can see how it almost appears to be common wisdom.  But have we got at the truth here?  Untested ideas such as this are the very kinds of potentially wrong hweuristics that ethnographic research is designed to correct.  If you ask students what they want, they will be happy to tell you, but they might provide the wrong solution.  Asking students for their “solutions” is the design charrette approach.  Do you want soft chairs or hard chairs?  Do you want this pink or blue?  Do you want all your courses to have the exact same layout?

I think the more successful approach is participatory service design.

The first step in participatory design is ethnographic research – close observation in a systematic and rigorous way to find out how students interact with courseware.

The second step is rigorously interpreting the data you gather, and applying it to the design problem in order to arrive at an optimal solution.  Interpreting the data and evaluating  potential solutions is done by people whose profession involves service design.

There are many stakeholders in the selection and support of courseware, and one or two platforms on campus will often be the right choice in balancing across constituencies.  And even though that solution may quiet student complaints, it doesn’t address what I’m guessing (and what ethnographic research could determine) is really the source of student complaints — poor design of online learning environments.

I take a lot of classes so I am frequently in the role of student.  But I am not suggesting my perspective leads directly to “the solutions”  for design of online courses.  Being a student just helps me understand why participatory design succeeds. (IT people sometimes call this “eating your own dog food,” meaning that putting yourself in a user’s shoes helps you design better.)

As a student, online courses can annoy me in 3 ways.  There are many, and in my opinion, better solutions to these annoyances than herding everyone into the same courseware corral.   (I’m only looking at a student point of view here; I’ll briefly mention faculty-as-users in a bit.).

My first annoyance is that it is often hard to figure out the navigation of online courses — where has the usual “stuff” been put.  Not everyone takes the same meaning from labels like “syllabus” or “course policies” or “assignments.”  An institution or program could, of course, “design once” and make everyone fit into the same straitjacket — same courseware platform, same course design, same labels, etc.  Then students just need to learn that one pattern. It isn’t hard to see that there will be a downside to this approach, even if it might smooth things over with students in the short run.  Personally, I’d prefer good designs customized to course objectives over sameness.  Some people get course navigation right — it’s like walking into a room where the furniture is arranged nicely versus walking into a room where things are awkwardly placed.  It’s essentially a design question, not a platform or technology problem. So, how costly is it to raise institutional knowledge of course design?  Or, how costly is it to sidestep it?

Another issue for me, wearing my “student hat,” is understanding the affordances of a platform.  Can I plz be told how the tool works in some plain simple way?  If the lecture videos can be slowed down, for example, that’s important to know. Can I run my writing thru a check that will help me improve it prior to turning it in?  Again, research can tell us how best to present explanations so that students take advantage of features and avoid pitfalls.

Annoyance number three involves knowing what I have to do each week.

This is presented in wildly different ways in each course I take – it is not dependent on the platform, it reflects the instructors’ preferences. In the worst cases, weekly lectures are in one place, assignments for submission are described in another, the requirement to post to discussion forums is listed somewhere else.  And as these things inevitably change during a course, instructors sometimes update deadlines in one place but not in all the places where the information is duplicated.  The result is significant administrative overhead for students madly trying to complete weekly requirements.  The problem is not complicated — it’s just this:  the design does not efficiently allow the student to “know” when they are done for the week.  The corollary is inefficiency for the student trying to plan ahead for what they need to do.

I suspect some of these design flaws are actually features for faculty — they are done to minimize the instructor’s administrative overhead either during the course or in migrating the course forward for the next semester.  This is the reason that participatory service design needs to study all users of a system — how do the faculty use courseware?  what about other stakeholders?

My overall point…   Yes, one or two courseware platforms on campus may well be necessary for a host of reasons, but I’m not ready to interpret student complaints as buttressing a “one-size-fits-all” decision about instructional design.   For me, student complaints may lead in a different direction.

One of the IT speakers I heard at ELI a few years back spoke to the problem of balancing standardization and innovation, questioning if short term gains from standardization are outweighed by creating inflexible people in a rapidly evolving climate.  It could be that the really strong institutions of the future will have devoted as much or more time to raising institutional knowledge about the design of learning environments as they do to standardizing on pre-selected platforms.

                                                        –Karrie

Tools for organizational effectiveness

I believe quite a lot of people–even with no background in operations or manufacturing–could set up an assembly line that delivered a reasonable amount of efficiency.

I believe the skill required to do that is based on knowledge that has become part of our culture.  Through our ordinary, everyday living and working experiences, we have somehow learned how to organize a productive process that is sequential, linear, modular and cumulative.  And part of our cultural knowledge is knowing when an assembly-line approach is appropriate.  We know almost instinctively how to take advantage of the particular affordances of that kind of process for volume, efficiency and quality control.  We seem to quite naturally understand the difference in affordances between the home-based workshop or skilled artisan, and the collective enterprise that uses the assembly line approach.

What does the future hold regarding our cultural knowledge of productive processes?

Assembly lines are useful in some situations, but we’re increasingly working in a world that needs knowledge workers–handling big, messy, unstructured problems. Knowing how to put together an assembly line is of limited usefulness.

I think the tools of production we need now, and which should seem as easy and natural to us as the organizational tools of the manufacturing era, are tools like agile project management (or even standard project management), change management, establishment and leadership of high performing teams, knowledge management, fostering learning organizations, understanding and nourishing innovation, service design, program assessment, and so on.

We all spend probably quite a lot of time and effort acquiring skills in these areas.  Even so, sometimes it’s hard to get a lot of traction in your workplace for putting your newly acquired skills into common practice.  Legacy “best practices” and “common wisdom” carry cultural weight even when they are no longer yielding the results they once did.

Just as the successful enterprises of the manufacturing era made an asset of their ability to use processes appropriate to their sphere of work, I think organizations today will succeed or stumble based on their ability to make use of the appropriate process, design and management tools.

And that our grandchildren will find it a simple matter to establish and launch high performing teams…                                                                                     Karrie

Ivies Plus Meeting at Yale

My first visit to Yale University was Friday, Oct 19, for the Ivies Plus Meeting of librarians.  Quite a lot of beautiful stone buildings and a fair amount of scaffolding and reno work going on to maintain those beautiful buildings!

The meeting focused on undergraduate students.  The first talk was by Susan Gibbons, University Librarian at Yale, on “Enhancing the Undergraduate Library Experience.”  She spoke about the well-known anthropological work done at Rochester, and some about Yale.

Gibbons is a wonderful speaker, zeroing in on the teaching and learning rationale behind innovations.  For example, lots of libraries have adopted cafes or coffee shops within library space “so students don’t have to leave the library.”  Gibbons pointed  to the key teaching & learning reasons for a cafe – designing it to foster the kinds of conversations that enrich and deepen student learning.  She pointed to the coffee shops of the 18^th century as great places for intellectual exploration, and talked about the way comfortable coffeeshops near campus were places where students were intermingling even with faculty and discussing their work.  I’ve certainly seen a lot of library cafes in the last decade.  Some café spaces really foster those conversations, and even attract faculty as well as students.

Designing spaces to foster a community of scholars was a thread throughout her talk.  She told a story about a physics library where group spaces were set up with whiteboards – not only were students gathering there to work on problem sets, but because of the open set up, faculty and grad students would sometimes wander by and volunteer help to students who needed a little extra explanation or coaching.  I’m fascinated by these examples of faculty being in the library…  Again, I would say the key to this innovation is not just “lots of whiteboards” but the activity and interaction that the overall design fostered.

Gibbons briefly pointed to the value of photos as a way to get student answers to questions – something I would like to copy.  Her example was having students take pictures of where in the library they felt lost.  Of course, the stacks were highly featured in these photos, and  in response the library enlisted fellow students–known as stacks ninjas–to help as guides.

She also pointed out how students don’t feel necessarily welcomed or included – and that the library is a welcoming space (or can be) because it belongs to everybody.  It’s neutral.  Her discussion of ethnography also made me think of how students might think they are unwelcome, or be somehow less likely to use, library spaces that seem to be for some particular purpose.  Many libraries I have worked in have lots of “rooms” that seem to be special purpose even though anyone is actually welcome to study in them….

Gibbons highlighted the value of ethnography in breaking down wrong assumptions, such as the idea that students sleep in late.  As has been repeated many times, the initial ethnographic studies showed students were up and about their busy lives early in the morning, often carrying everything they would need for the day, all day long.  And that because they are so busy, they don’t start studying or research work in earnest until late at night much of the time.  Libraries I have worked in have yet to really shift their services substantially toward late evenings….

Gibbons also noted that furniture on wheels not only allowed to students to design optimal work spaces flexibly, but it also allowed the library to quickly reconfigure spaces for different kinds of events.

Gibbons said that in designing collaborative work spaces in the library, it’s helpful to know what spaces on or near campus people already use in that way, and why they like those spaces.  She also remarked that good students can be fairly intentional about the way they “design” their study situations – for example, not bringing a laptop cord, knowing that they then have to buckle down and study before their battery runs low.  Or finding particular spaces in the library that are free of distractions (such as poor internet connections).  Her overall point was, of course, that there are a plurality of study/research space needs even for a single student, and that we need to look at our buildings to see if we’ve provided in a balanced way for these multiple contexts.

A big part of Gibbons’ talk had to do with the way universities have separate administrative units to handle aspects of what seems, to students, like a single messy problem.  Students experience their work as the intermingling of technology, writing, and research – but support is not offered seamlessly for these areas. I think libraries generally know this by now, although it still is a hard problem to solve.

At Rochester, evidently, some librarians were trained as writing tutors, and the writing tutors spent some time at the reference desk.  That seems like a great step forward in reducing the intellectual distance between these services if not the administrative and geographical distance.  Especially in places where a good part of the library’s ability to reach undergrads is through a first year writing course.

Gibbons and the next Yale speaker, Emily Horning, talked compellingly about their Personal Librarian program and how – given the context of Yale and the key features of campus life for freshmen – this was a successful program.  There is a strong identification with one’s college, for example, and that provides a good way for personal librarians to connect with their students.  The actual service uptake is 15-20%.

One of the common themes throughout the day, for me anyway, was the way all these places are leveraging student workers in new ways.  The stacks ninjas were one example (they had a real sense of camaraderie, it was clear from the photos, and had designed their own Tshirt), but several people I spoke to were using students throughout orientation to promote the library.  The student workers were seen almost as advocates in some instances – part of their job was promoting the library.

There was talk throughout the day about the late hours students keep.  One “library opening” after a renovation project happened at midnight and drew a huge crowd of students (the food, if I remember, was gone in 12 minutes …)  Emily Horning also spoke of “late night study breaks” being offered for relaxation during crunch time, realizing that even at that late hour – 11pm or so – students were really just getting started with their studying.

The Yale librarians clearly applied ethnographic research methods and thinking to their own context.  I wonder how well the rest of us would do in answering the question “What are the key defining features of undergraduate life on our campus?” as we seek to develop our services and products?

All in one – my Coursera experience

I’m not going to repeat my extravagant praise for Coursera and online learning here – it’s all in my earlier post (My Brain on Coursera).  This post is my thinking about online learning from an instructional design perspective, and helps me capture my thoughts for my own experiments in online teaching and learning.  I expect to be using online environments for library instruction, and this helps me remember things I would want to do differently while they are fresh in my mind.

I was challenged by a colleague to think about what I would be willing to pay for as a student, and of course we are all thinking about the ways the library might contribute to further developments.

Suggestions about online course web sites

Managing Weekly Progress

• a straightforward way (one single place) to know what I must do, how long it will take, easily see that I have completed all steps for each week
• use actual dates rather than the nebulous “week one” and “week two” labels

Improve Overall Course Navigation

• separate course elements into discrete categories so they are easy to find (how things will work or what I call reference material;  video lectures; notes; review materials for tests; etc)
• create a printable course calendar that shows the deadlines  for the entire course.
• enable the course calendar to integrate with all my other Coursera courses if I choose
• create email or other alerts so that I re-visit the calendar when there are changes

Expand ancillary learning materials and create more learning reinforcement

• include more exercises such as glossary exercises, practice questions, etc
• let me see “great examples” of writing about the concepts.  I don’t mean sample assignments, since I think people would tend to just conform and copy what they see, but interesting approaches to either to sub-parts of an assignment or to related assignments given in the past.
• every unit needs a more extensive, non-graded quiz.  This could be started by allowing students to contribute quiz questions, which might double with their “muddiest point” confusions ( see below).
• use “muddiest point” questionnaires to direct students to appropriate ancillary materials (perhaps even just “voted up” content from the discussion forums)
• all math examples or simulations should be separated out for repeated replay at variable speeds, and these should be a priority for development of ancillary materials, esp additional examples of the same concept or different applications of the concept

Improve peer assessments

• let me grade an assignment specifically created for peer assessment training and see how my score stacks up against a standard (this was mentioned as something Coursera is doing but it wasn’t used in my course)
• create more of a template for peer graders to apply the rubric
• allow for a pair conversation as an optional activity.  I agree to view someone’s assignment before we submit, and make suggestions to them.  And they review mine and make suggestions to me.

Raise standards of grading by allowing for revisions

• to prevent grade inflation which I think may be happening (graders are reluctant to give bad marks), allow anyone who has gotten a very poor grade to do revisions, if there are enough volunteers in the class who got good grades to do a couple extra peer assessments.

Use peer assessments to inform teaching

• let peer graders fill out something that indicates what they think students don’t get.  Perhaps a formula sentence like “I think many students may still be unclear about _____” and choose from a list, with the option to fill in a word of their choice.

Expand the Community TA concept

• Allow me to seek a tutor or barter an exchange – I will give feedback on writing quality in exchange for help with a statistical concept. Devise it as a low stakes exchange by limiting it to one week of conversation (in case it doesn’t go well).

Fix the “mosh pit” problem in the discussion forums

• Let me fill in a profile so that I become assigned to be a member of a group where the discussion board is going to be more at my level of discourse
• Let me further refine my ability to engage in ongoing discussion with just the people of my choice – no more mosh pits with 20,000 undifferentiated people.
• Create more structured threads including some short answer Q&A threads that would serve as review and test preparation materials.

Improve tests

• Students want to feel that the test questions got at the most important concepts, and I can’t say I felt that  about my test experiences.  The preponderance of questions toward some topics seemed odd.
• When the right answer is not grammatically responsive to the question, I find it really confusing.  When it doesn’t sound the way a normal person would answer the question, I have a hard time understanding it as the right answer.

Include critical thinking about and evaluation of resources

• Develop at least one unit around the idea that all resources on a topic are not created equal, that we have to evaluate information.  Most faculty have at least one example of poorly presented information or misguided “popularizing” approaches to a topic.
• Use characterization and summarization exercises (these are hard for students to do)
• Clearly label all resources for citation – lectures, guest interviews, etc.  These are learning objects and need a bibliographic status!

MOOCs – What I Would Pay For?

1. A proctored exam in order to get a more meaningful certificate.

2. The ability to download the lectures more easily and quickly to a device, so I can engage with the class in offline situations.

3. A way to find and stay connected with other top performers in the class.

4. A simple way to receive access to the optional materials that I want – like checking off a list of what I want and making one payment, and bingo, it all arrives by some timely date.  (Like a coursepack.)

Library Involvement in MOOCs:   What might we do?

Collaborative assistance in developing the “critical thinking and evaluation of information” unit in the courses.

Assistance with discovery, supply, organization (metadata) and management of copyright-free educational materials.

Assistance with the flow of optional materials to students who paid whatever price was determined.  (The coursepack idea).  [We need to get there first – publishers are actively looking for pay-by-the-drink models and the tens of thousands of potential consumers in MOOCs will be on their radars.]

Assistance with the ancillary learning materials, as they are rated more or less useful by students.  This might involve metadata, search optimization, recommendation systems, categorization not only by topic but by level of sophistication and complexity, presentation interfaces.

Assistance with the development and production of interactive textbooks or learning objects by Penn faculty, using technologies aimed at simplicity and sustainability.

And for on-campus students in blended learning…

Library instruction to help students develop information skills within a discipline can be included in blended courses aimed at on-campus Penn undergraduate and graduate students.  We can and should transform the typical instruction we do in this area.

Faculty and librarians have identified many of the skills gaps that cause students to do poorly in school work based on library research, ranging from developing a good research question to comprehensively exploring relevant literature for a lit review.  Exercises to help students overcome those skill gaps can be designed in an online “computational” environment just as well as exercises for mastering course content.  In other words, we can use computational methods to improve on the one-on-one or classroom-based patterns of traditional librarian teaching.  We can incorporate learning science advances such as the ideas around mastery and the zone of proximal development, for example, which would be next to impossible in the kind of face-to-face teaching we normally do with students.

Freed of the huge operational demands of repeated delivery of library instruction, we can devote our knowledge and abilities to the design of very high quality instruction.

–Karrie Peterson

Coursera – Werbach – Gamification for the Enterprise

Kevin Werbach’s last set of lectures on gamification for the enterprise are flat out fantastic.  By gamification for the enterprise, he means gamification directed at improving organizational effectiveness (rather than customer-focused business objectives like increasing sales, etc.)

His examples are from well-known businesses, and  range across:

• HR-type onboarding and training processes
• Knowledge management (WhoWhatWhere at Deloitte – cool!)
• Incentivizing better customer service in various ways, such as encouraging idea-sharing or gamifying the way individuals learn better practices
• Increasing systems thinking about use of professional development funds
• and beyond…

Wow.  Through this course I am realizing how the commercial sector is already integrating gamification into their success in very sophisticated ways.

Over the years, I’ve seen some pretty rudimentary and ham-handed attempts to turn information literacy skills-building lessons into actual games–the gamification class taught me why these didn’t click.  These efforts are not gamification. They were/are infotainment, i.e. an attempt to sugar-coat failed pedagogies.

But back to organizational effectiveness….

Gamification techniques strike me as transformative for internal organizational effectiveness.  Why?

It would be wonderful to have a staff comprised 100% of deeply self-aware people with highly developed motivation to learn from feedback, to solicit criticism of their products, to do the “after action review” for projects that help the organization learn rapidly and broadly.  But that level of self-awareness is pretty difficult to achieve all-day-every-day in real life.  We aren’t all perfect.  And even if many of our colleagues are willing to learn from mistakes, it won’t help if the organization’s culture doesn’t consistently support the kind of openness that fosters learning from mistakes.  Fail early and fail often is a mantra I hear at conferences, but much harder for organizations to foster that culturally.  So, maybe designing our organizations to learn productively using gamification would get us where we want to be a lot faster?

I have thoughts about a “game approach” to help develop project management skills — something that would foster self review by teams and reward feedback loops during projects.  It would be a formative approach, rather than a post mortem that is so dreaded it is often bypassed.

Of course, the examples Werbach draws from business seem somewhat out of reach for academic libraries.  Corporations can fund this kind of organizational effectiveness effort–requiring both design and implementation expertise–with part of their profits.  If you want people to learn before mistakes happen, using gamification to direct staff behavior, you need an informed, articulated program of target behaviors.  Presumably managers could define these targeted behaviors, although I think in actuality many managers would find this harder than expected.

What should we do, how much of our slender operating/innovation funds can be devoted to this kind of organizational re-design?  Among all the risky things libraries need to experiment with in order to adapt, anticipate our users, and offer impactful services, where might gamification for organizational effectiveness fit?

–Karrie Peterson