Non-Traditional Scholarship and Peer Review

Picture of Titanic sinking in ocean with caption "It could be that the purpose of your life is only to serve as a warning to others."
This “Mistakes” poster available on introduced John Unsworth’s talk. The caption reads, “It could be that the purpose of your life is only to serve as a warning to others.”

This is a brief summary of takeaways from a Scholarly Communications Symposium on the topic of peer review, held at UNC Greensboro on April 2, 2018, based on 16 pages of notes typed on the fly; any errors are my own, and corrections for accuracy are welcome (sylvia dot miller at duke dot edu).

Click here to jump to the main sections of this post:

Digital Humanities – Community Engagement – Transdisciplinary Research and Education – Last but Not Least: What Counts

Three flavors of non-traditional scholarship were addressed at this valuable all-day symposium: (1) scholarship characterized by digital methods and outputs and (2) engaged scholarship, i.e., public humanities and the work of scholar-activists in communities outside of the academy; and (3) transdisciplinary research and education. The focus, however, was mostly on the first two, because neither is easily represented in the form of written narrative that traditionally flows into the kinds of publications that tenure-review committees prefer to evaluate, whether by force of habit, unwillingness to change, lack of appreciation for the new work, and/or absence of established evaluative criteria. The symposium aimed to provide a way forward by sharing practical examples in which peer review of non-traditional work had been successfully accomplished.  Takeaways from the three individual talks and two panels that might be instructive follow.

Digital Humanities

Both Andrew Torget, Department of History and Digital Humanities Lab, University of North Texas, and John Unsworth, University Librarian and Professor of English, University of Virginia, candidly related stories–long, detailed, complicated, and even dramatic–about their struggles to gain recognition for their work.  Here I have attempted a mashup of lessons they imparted:

  • It is a special challenge to have digital scholarship evaluated in the medium in which it is best understood and appreciated; ideally, additional outside reviewers would be added to the tenure-review faculty board who are experts in the medium. In one case (Andrew Torget, University of North Texas), five additional outside reviewers were added to evaluate the digital aspects of his work.
  • A practical if imperfect approach for reviewers who want to see a traditional portfolio is to write a précis for each project, including careful delineation of the candidate’s role in collaborations as well as links to the digital work.
  • Respect for new work cannot be gained all at once; winning grants, speaking at conferences, and having one or two vocal faculty and/or administrative supporters are helpful. Added to these, ongoing meetings and conversations to share the work can familiarize colleagues with it along the way and cultivate relationships. Humility (“I need your help”) is effective.
  • Framing digital projects as nodes on the way to a book can be effective, if that fits the researcher’s plan.
  • Once a scholar working primarily in digital media gains tenure in a department, it is easier the next time, for the next candidate. Nevertheless, the first time should not be an exception to the rule; to move the department, the institution, and the profession to a new place for future scholars, the tenure guidelines must be changed.
  • While statements from societies such as AHA and MLA have a positive influence, changing guidelines locally is a monumental task (in one case, it necessitated a meeting of 35 people). The goal is to list “digital projects that have been favorably reviewed” on par with books and articles as work that will satisfy the requirements.
  • Both speakers expressed awareness that all of the above might have been even more challenging if they were not white males.

Unsworth concluded, “In universities and fundamentally traditional humanities departments, change is so slow that it could be mistaken for stasis, unless your frame of reference is decades. . . . The digital humanities have changed what we write about and have started to change what we consider an interesting research project.”

Community Engagement

Emily Janke, Director of the Institute for Community and Economic Engagement and Terri Shelton, Vice Chancellor of Research and Engagement, UNC Greensboro are working with a task force that includes James Albright, Director of Guilford County Emergency Services, to write “community-engaged guidelines.” The panelists argued that quality is validated by long-term relationships, ongoing contracts with local organizations, and tangible measures such as fewer deaths and safer streets in the communities where they are working to reduce gangs and domestic violence. In ongoing community work, evaluation and implementation inform each other iteratively and happen simultaneously; legally approved structures form that can make it easier for government agencies, communities, and academics to collaborate on other projects in the future.

Guidelines will place value on open data, not only for ease of community access to information and services but also to enable reproducibility and scalabiity.  Later in the day, the theme of openness was central to a panel on community-engaged scholarship given by Stephen Sills, Director, Center for Housing and Community Studies, Somya Mohanty, Associate Professor, Department of Computer Science, UNC Greensboro, and Evan Goldstein, Assistant Professor, Department of Geological Sciences, UNC Chapel Hill.  The audience was especially taken with a rainbow chart of ways to make one’s workflow more open (by Jeroen Bosman and Bianca Kramer of Utrecht University).

Sills said that his department (Sociology) is rewriting their tenure and promotion guidelines to include community engagement in the category of scholarship.  They are trying to write the guidelines in a way that is open and flexible; he mentioned “scholarly products” as a potentially useful term that might include, for example, a documentary with a million views on YouTube as well as scholarly articles.

Transdisciplinary Research and Education

Sharing exciting examples about new anti-microbials, “smart plants,” and prosthetics, to name a few, Daniel Herr, of UNC Greensboro’s Joint School of Nanoscience and Nanoengineering, emphasized the importance of encouraging the convergence of disciplines (such as  nanoscience, physics, math, chemistry, and biology) and entrepreneurial creativity.  Herr did not address tenure and promotion directly but indicated that the quality of the work is evidenced not only in publications but also patents and inventions that benefit society.

Last but Not Least: What Counts

 Opening the day–but shared here as more of a conclusion–was a thoughtful talk by Kathleen Fitzpatrick (MSU Director of Digital Humanities and previously Director of Scholarly Communication for the MLA), who zoomed out to offer a larger, more philosophical view of the challenge. Her talk was about what we value:  what counts, what should count, and why we should give up our reliance on easy quantitative measures to seek more subjective, qualitative judgements about a scholar’s path to intellectual leadership, from establishing one’s voice, to helping other scholars establish their voices, to having an impact on his or her field.  The tenure process, she argued, is and ought to be individual; we need to seek ways to be equitable without imposing an impossible objectivity.

To consider “what we genuinely value,” she warned against the following:  being stymied by the unfamiliarity of the work, trying to turn a tenure candidate into someone different from the person who was hired, doubling the candidate’s workload (adding traditional on top of non-traditional work), relying too much on quantitative measures, disqualifying reviewers with whom the candidate has presented on a panel or served on a board (thus punishing collaboration and professional relationships), marginalizing teaching.

She advocated for the following: evaluator learning (including support of junior faculty as they “mentor up”), engaging with the work on its own terms in in its own medium, recognizing distinct measures of impact across fields, engaging appropriate experts to evaluate the work but not relying on them entirely, balancing objectivity and subjectivity, rewarding collaboration.

“Standards and processes should be considered in light of the ways the work is being read or experienced, how scholarly values are manifested in a career in process,” she concluded.

Her slides are available here.

Resources she recommended were society guidelines and the policies at Emory University and the University of Nebraska-Lincoln.


Reflections on DH2017 Montréal

Room in Museum of Fine Arts, Montréal
Video projections of swaying trees and twittering birds in the Montréal Museum of Fine Arts bring back the magic feeling that painted clouds and cherubs might have inspired centuries ago. Photo by Sylvia K. Miller.

By Sylvia K. Miller and Hannah Jacobs

Some new trends emerge as Duke colleagues reflect on DH2017 in Montréal and contrast it with previous DH conferences. Following is a distillation of our observations in the form of a list of takeaways:

  • Collaboration. There was more frequent reference to project partners, indicating more acceptance, even embracing of, collaboration. Many presenters explicitly, fully, gratefully thanked their team colleagues, showed their pictures on the screen, etc. Associated with this positive development for DH and humanities at large is concern over how collaborators are to be formally credited for publication and tenure.
  • Diversity. The diversity workshop, for the first time open to all attendees, was a critical moment indicating that the community remains committed to addressing its diversity challenges.
  • Librarians are more and more taking ownership of their role in enabling and supporting DH. I attended the “DH in Libraries” SIG meeting, and the lecture hall was quite full; I’d guestimate 200+ people. There were a number of projects to classify library/archive collections in new ways.
  • TEI is more synonymous than ever with XML in the library world and world of DH scholarship. This is important for publishers to know if they are collaborating on multimodal projects that will have integrated library or repository components. (XML comes in different established schemas; TEI=Text Encoding Initiative.)
  • “Phygital.” I heard more about the interplay between analog and digital forms, particularly 3D printing. (One paper called multimodal physical/digital projects “phygital,” though it is impossible to predict whether the term will stick!) The phenomenon of 3D printing of historical artifacts is fascinating and problematic: is it right to “copy” an artifact before returning it to the community from which it was stolen? (What if the artifact is a human bone?)
  • Music/art. There was more art and music this year, though the emphasis of the conference is still more computational than artistic.
  • Visualization. The expansion of visualization as an accepted DH practice stood out. There was an increased presence of visualization in its many forms; especially prevalent were node maps showing network relationships and geo-maps showing data sources or trajectories. Some of this work was also self-study (e.g., Taxonomy of Digital Research Activities in the Humanities, TADIRAH).
  • Cultural heritage work was a theme. In Europe it is well funded, but systems developed there are not universally applicable; scholars working with American Indian communities find that digitization challenges and metadata schemas are very different in the US context.
  • Overlaps. Several different teams around the world found that they are developing similar tools. At DH2015 in Sydney, there was a lot of special OCR work and machine-reading of historical texts with special scripts or other visual challenges. At DH2017, there seemed to be a significant amount of attention to adapting face-recognition software to recognition and comparison of other types of visual data (e.g., newspaper layouts, furniture/decorative arts, etc.).
  • Publishing. The conference demonstrated a wide variety of computational, visual, and interactive scholarship that demands new modes of publication. Sylvia has noted some specific, if scattered, developments elsewhere; taken together, they remind us that scholarly publishing in its current form is seen by practitioners of DH as a barrier to growth and innovation.
  • There was some AR and VR at the conference but not as much as one might expect. One exciting AR project stands out in retrospect, a project by Amanda Marie Licastro (Stevenson University, MD) to share the experience of Syrian refugees and create empathy in millennial students, who are, according to research, 40 percent less empathetic than earlier generations(!); in the Q&A session, she told a striking story about a student whose views were radically changed by the VR experience. I also heard about an exciting visualization project from Rachel Hendery of the University of Western Sydney, a projection of a node map inside a dome, where people can experience it together in 3D without having to wear 3D equipment.
  • Funding. Mellon and NEH are the important funders of US work. European work is well funded by governments and the EU.  There is a new government-funded DH initiative in South Africa.
  • Peer review. There were many calls for proper peer review and recognition of DH as legitimate scholarship for promotion and tenure. One speaker mentioned that the University of Florida is the latest institution to create a DH peer review committee; content creators were asked to recommend a level of equivalency in terms of traditional scholarship on which the DH project should be evaluated.  This was considered not ideal but perhaps a necessary interim step. (In this context, the Mellon-funded project at Brown University to evaluate digital scholarship at the department level seems especially interesting and ambitious.)
  • Congenial spirit. There was a great deal of friendliness and mutual appreciation at this meeting; many scholars, especially graduate students, feel alone at their own institutions and appreciate meeting like-minded people. The group Tweeted up a storm! See #dh2017  (and @SylviaKMiller)
  • Attendance. I heard that there were around 1,100 people at the meeting.  I have not been able to confirm this online and have written to the ADHO–Alliance of Digital Humanities Organizations, the umbrella organization that hosts the conference.
  • Future meetings. Next year the meeting will be in Mexico City, June 24–30, 2018; in 2019, Utrecht.