National Network of Libraries of Medicine
nnlm.gov
Archives
This page was archived on: May 02, 2008 | View page metadata
Document content is not current. Links may be broken.
| National Network of Libraries of Medicinennlm.gov |
Archives |
|
This page was archived on: May 02, 2008 | View page metadata | Document content is not current. Links may be broken. |
||
Mark P. Haselkorn
This paper presents perspectives from the field of technical communication on the planning and evaluation of information outreach. As described by the National Library of Medicine (NLM) project for which it is written, the intent is that the perspectives presented here will be integrated with the perspectives of many other relevant fields to help produce a general guide to the conduct of evaluations of medical information outreach.
I begin with an historical overview of the field to help the reader understand the particular perspectives of technical communication. I then address the specific items requested by the NLM project: (1) input and output variables, (2) process variables and interventions, (3) evaluation methodologies, and (4) models.
Technical communication is a relatively new field. It grew out of efforts during World War II and the Korean War to introduce new technology on a wide scale to non-specialized users-the soldiers. These efforts revealed a profound gap between the functionality and complex information associated with technology and the ability of people to access, understand, and benefit from that functionality and information.
In the military, instruction was largely face-to-face, word-of-mouth, and hands-on. Training was an apprenticeship based on watching and doing. Quite complex material can be-and was-passed on in this way, but learners needed time to absorb it, and during the war, time was lacking. In addition, the large number of people who had to learn, compared to the few number of trainers, made apprenticeship impractical. Somehow, information had to be transferred in other ways.
Initially, the military turned to the engineers and scientists who had created the new technology to provide the training and documentation essential to ensuring that their equipment could be properly used and maintained. Unfortunately, this did not work out. There were a number of reasons for this.
Most obviously, there were insufficient numbers of technologists, and they generally lacked the communication and training skills needed for effective information transfer. Since many of the American recruits in WW II had little more than an elementary school education, the technologists' job was even more difficult. In many cases, their intimate knowledge of the technology made them particularly unsuited to recognize and relate to the problems of novice users. In addition, those with technical and scientific knowledge saw the creation of new technology as the real work; the introduction and maintenance of that technology was a post-effort detail. Finally, in the 1940s there was no field that studied precisely this problem of matching user needs and abilities with the design and use of technology. This gap between industries' ability to produce technology and the military's ability to successfully introduce it often resulted in poorly trained troops and under-utilized equipment. (Korea was a particularly painful example.)
The rise of interest in and use of technology after the war led to the growth of technical writing as an occupation. Initially, this occupation was performed by writers who had some technical knowledge and by technologists who had some writing skills. These technical writers were generally not well received by scientists and engineers deeply involved in technology development. In some cases, technologists failed to see the value added by technical writers, but more often, technical writers were forced to work under conditions that made it difficult for them to add that value. User information was not even on the critical path of product development, and technical writers began their work late in the game, almost as an afterthought. No wonder engineers and scientists saw them as an annoyance. These writers were forced to pester them with seemingly simplistic questions just at the time when they were struggling to get the product out the door.
During the early stages of a technology-based industry, users are specialists and technology sells itself. In the computer industry, a new chip was a new product. As an industry matures, however, it attempts to reach a larger, more general market, and then it begins to shift its focus from technology to the user. As the computer industry matured it found, like the military in World War II, that there was a profound gap between the functionality of their products and the ability of non-specialist users to access and benefit from that functionality.
In response to these issues, the field of technical communication was born and grew. "Communication" replaced "writing" because so much more than writing was needed to address the gap between technology and user-interface design, creation of online help, usability evaluation, use of multimedia, and design of electronic information. Today, the problems associated with providing valuable information to the end user still exist. In fact, they are more pressing than ever as the computer industry in conjunction with other related industries moves further and further towards serving a mass market. Against this background, technical communicators still struggle to define their role in an increasingly vital area of the information industry.
Given the focus of this project on medical information, it may be helpful to consider another field which grew up at a similar time and for similar reasons as technical communication-the field of rehabilitation medicine.
Like technical communication, rehabilitation medicine received its initial impetus from WW II and took a number of decades to gain maturity. In the aftermath of World War II, the large number of veterans returning to the U.S. with catastrophic injury put extreme demands on available medical resources. Traditionally, surgeons had treated these injuries and supervised aftercare and rehabilitation (or had not recognized the need for it); but with the number and complexity of war-related injuries, surgeons alone simply were not equipped to deal with the wide range of human problems associated with the actual injury itself. The surgeon was, in a sense, the medical technician. He was concerned with repairing broken bones and reconstructing human tissue. Issues of pain or psychological trauma went beyond the scope of most surgeons' expertise and interest. They generally did not address whether or not the patient would be able to return to work or brush their teeth. The wide range of problems associated with disability, combined with the sheer number of injured soldiers, necessitated that a new specialty be developed to address this situation.
But when rehabilitation specialists began to surface, utilizing interdisciplinary teams and trained in more holistic issues of patient recovery, the surgeons were initially suspicious of this invasion of their "turf." Why bring in a specialist to do what surgeons had been doing successfully for years? Gradually, however, surgeons began to realize that having such specialists enabled them to focus on what they did best; namely, the physical repair of the human body. Rehabilitation specialists did not replace surgeons, far from it. Because surgeons were relieved of this aspect of a patient's recovery, they were actually able to perform their job better. Today surgeons and rehabilitation medicine specialists work together on interdisciplinary teams that treat and manage disability.
Analogous to surgeons are the engineers and programmers who work with hardware and the internal logic of the machine. Surgeons can set a broken bone or perform live-saving operations, but they tend to be uncomfortable and untrained in treating what comes after-namely, their patients' pain and ability to function at home, work, and in the community. Similarly, computer engineers and scientists are skilled at creating innovative, cost effective, and reliable technology but they have less interest and training in communication skills, user psychology, and workplace practices. Rehabilitation medicine addresses the more human, holistic issues of patient recovery, thereby freeing surgeons to do more of what they do best-surgery. Similarly, technical communicators focus on end-users and their relationship to the technology, thereby freeing engineers and computer scientists to do more of what they do best-making the most efficient and reliable technology.
In both these cases, technical communication and rehabilitation medicine grew out of the need to address larger human issues surrounding a more technical practice. In each case, someone was needed as an advocate of human aspects of the technology. Also in each case, technologists were initially hostile to the encroachment of "non-specialists" (or specialists in more human-centered areas) but eventually came to see the newcomers as partners who could help them do their job better.
Given this background, technical communication can be seen as primarily taking the perspective of "user advocate" in the life cycle of information and information technology. The primary activity can be called "user-centered design." This perspective originally was applied at two points: (1) content-the creation, design and maintenance of the information itself and (2) delivery-the interface between user (reader) and information technology. Over the last decade, however, this perspective has been applied further and further back in the process, so that now it is not uncommon to employ user and task analysis as the first step in developing an information product. The goal is describe the "look and feel" of that product which can then be used as a guide to evaluate success throughout the stages of product development.
This is a key message that technical communication brings to the NLM project-evaluation should not be applied only at the end of an informational project; post-project evaluation should be tied to an initial definition of goals or requirements and then to an ongoing evaluation of success throughout the development phases.
What, then, are the relevant entities to a technical communicator? (It should be mentioned that we do not talk about things like input and output variables, but I will attempt to put our relevant entities in the category terms supplied by the project.)
Input Variables: Not surprisingly, a crucial set of input variables for a technical communicator relate to the audience or user. In fact some technical communicators see the user as far more important than the information itself, the way it is presented, or the process of creating it. In other words, meaning can be seen as depending more upon what the user brings to their transaction with the "document" than what the information creator put there (Coney 1992). (Technical communicators use "document" to mean any form of information, for example a screen of information can be a document.)
Audience analysis in technical communication is rooted in the disciplines of composition and rhetoric (Coney 1987). Everything about the audience that can affect their interpretation of meaning is of interest. This means not just demographics (sex, age, etc.) but also experience (e.g. expert vs. lay), attitudes (e.g. level of interest in the subject matter, willingness to accept the intent), needs (e.g. simple instructions vs. considerable context) and group characteristics (e.g. homogeneous vs. diverse). Decisions about audience translate into process choices (e.g. writing to a single lowest common denominator vs. writing a document that can be used differently by different people).
Another key set of input variables for technical communicators relate to the purpose of the information or document. Purpose can be seen as being on a spectrum from informing (simple transmittal of information) running through persuading or motivating (which assumes informing) and culminating in doing (which assumes informing and persuading). Because of the practical nature of technical communication, most of the emphasis is on documents intended to help the audience do something (Mathes & Stevenson ????). For this reason, task analysis is an important component of purpose. By understanding the task from the perspective of the user, information can be designed to meet the user's needs, at the time they need it, within the environment and context of the task they are attempting to accomplish.
Closely related to the task itself is the task environment. It is not enough to design information and information technology that addresses a user's needs, it must also be designed to work in the environment that the user is in. This can be a physical environment (e.g. does it have to glow in the dark?), an organizational environment (e.g. will the user be multi-tasking while reading the information?), or an informational environment (e.g. must the information conform to and be archival within a specific document set?).
Information technologies themselves are input variables since they define and limit the many possibilities for presentation and delivery of the information. Even if analysis of audience and purpose indicates the need for a high degree of interactivity between user and information, this may not be possible if the technology is not appropriate. Information technologies also define what are cost effective options for information format and media (e.g. is video an option?).
While choice of media is best seen as a process variable, it is worth noting here that technical communicators are careful NOT to let information technology be the primary driver in making this choice. The decision to use text alone, or text and graphics, or drawings and photographs, or sound and video should be based primarily on what is needed (based on the analysis of audience, purpose, and environment) and only secondarily on what is possible. Gratuitous use of media (e.g. using video simply because it is available and new) is frowned upon as both ineffective and expensive.
Output Variables: One of the hottest topics in technical communication has been the search to find a set of statistics or metrics that can demonstrate the value of what technical communicators do (Redish & Ramey, 1995;
Carliner 1997). The search is difficult because many of the key "output variables," such as document quality or value-added, are extremely elusive to measure. Others, such as time spent producing the document, are relatively easier to measure but less clear as to what they mean (e.g. what is the relationship between time spent and quality produced?).
In the 1970s and 1980s, technical communicators made efforts to define clear, unquestionable measures of quality and value (Bandes 1986). Suggested "output variables" included comments on drafts (the hope was to establish a correlation between the number of comments and the quality of the product), conformance to requirements (this was an effort to mimic the industrial definition for quality of manufactured parts), and a wide variety of document characteristics that were put forward as being associated with quality. Today, technical communicators are less interested and less confident in all-purpose, unquestionable measures of quality and value. Instead, they focus on situational factors derived from an analysis of the particular project.
For now, I will present some of the "output variables" that today's technical communicators see as relevant. I will return to this discussion of measuring quality and value below under "evaluation methodologies."
Again not surprisingly, many of the relevant output variables relate to the user. User satisfaction is another elusive measure that can include overall impression, perceived relevance, perceived clarity, contributions to understanding or ability to perform, trust in the information, what was felt to be missing, what could have been left out, and other suggestions for improvements. Some measures of user satisfaction overlap with issues of user performance, such as how easily information could be found, search patterns, and ability to follow instructions.
User performance is another important area of outcome measures. There is a close link here between input and output variables since the key concern is whether or not the document served the purpose for which it was designed. If the product is informational or persuasive, then comprehension or belief will be the issue, but far more often for technical communicators the issue is whether or not the reader can use the information to accomplish the desired tasks.
Analysis of user performance is a complex activity and has lead to an entire sub-field known as usability testing or usability engineering (e.g. Dumas & Redish 1994, Nielsen 1994). One of the key insights of usability testers is that it is a highly situational activity with the variables of interest changing based on a variety of parameters (e.g. when the testing is done in the development process, changes in the conditions under which the user would perform the task). There is additional discussion of usability testing below under "evaluation methodologies."
When considering the value added to an information product by technical communicators (as opposed to the quality of a document), measures must go beyond the immediate user to consider the client organization both as customer and context. Just as outcome measures related to the user could be grouped under user satisfaction and user performance, so can outcome measures related to clients (organizations) be grouped under client satisfaction and client performance (Carliner 1997).
Outcome measures of client satisfaction relate to how the client felt about working with the technical communicators and how well the work was received. Probably the clearest indicator here is the willingness of the organization to re-employ the people and methods which comprised the technical communication approach, or to recommend it to other organizations. Satisfaction surveys and interviews are also used, generally with less clear results.
More significant and measurable is client performance. Redish and Ramey divide these outcome measures into increased benefits and reduced costs (see Table 1 reproduced from Redish & Ramey 1995). Increased benefits can include generating revenue (e.g. more sales), increased efficiency (e.g. more productive workers), and improved results (e.g. more people more efficiently taking advantage of the information product). Reduced costs can include reduced expenses (e.g. reduced cost of training or support or printing), saved time (e.g. less informational overhead), fewer errors (e.g. fewer user mistakes), and less risky operations (e.g. fewer accidents, less litigation).
Until recently, technical communicators have been more focused on input variables (i.e. analysis of the communication situation) than on output variables (i.e. evaluation of the quality or value added to the information product through their efforts). This is rapidly changing, however, as pressures increase on technical communicators to justify the value of their methods and contributions
The input variables discussed above are used to define a particular communication situation. These in turn are used to help make choices about many communication process variables. Output variables are also used to inform these choices in the sense that many technical communicators study the impact of process choices on the performance and satisfaction of users.
For example, the choice of appropriate style (itself consisting of numerous linguistic sub variables such as choice of person, word choice, sentence structure, etc.) is largely based on decisions about input variables of audience, purpose, and environment. Choice of style will also rely on studies of how style impacts reader response (e.g. Spyridakis ????), but again it is important to remember that the field of technical communication does not teach all-purpose answers so much as it teaches what choices have to be made, how to go about making those choices and the range of possible answers.
Style is just one example of a host of process choices about the look, feel, andstrategy of the information design. Other much discussed choices include the amount and relationship of instructions to contextual information-the minimalist debate (Carrol ????, Williams and D.K. Farkas 1992); the design of hypertext (Horton ????, Farkas ????); the role and levels of editing (Fran Buhler ????); integration of text and graphics (Williams ???), use of linguistic features such as advance organizers (Spyridakis ????); appropriate use of interactivity in a document, and various organizational strategies such as designing documents to be read beginning to end vs. piecemeal, designing information as a single document vs. as a member of a larger document set, designing a document to serve a single homogeneous audience vs. a set of diverse audiences, etc. Whenever information design choices relate to the creation of text, technical communication overlaps significantly with the field of rhetoric and composition.
Another set of process variables has to do with the nature of the role of technical communicators in the process of designing, developing, and maintaining an information product. These include when technical communicators are introduced in to the process (e.g. after the product is developed, during development, as part of the design team, during product conception); the organizational role and location of technical communicators and their relationship to other team members (Haselkorn 1989), and the organization of the technical communication team itself (e.g. how many and doing what?).
Yet another set of process variables involves choices of information technology and media. Sometimes these choices are constrained by a commitment to a given technology (e.g. designing an information product for Windows 98) but other times it is the communication situation which is the driver (e.g. designing a medical outreach product for minority audiences). In this later case, choices of technology and media should be dependent upon analysis of audience (e.g. what will they use? What will they prefer?), purpose (e.g. what best conveys the subject matter?), and environment (e.g. what do they have access to or can most effectively be introduced?).
The choice of a given technology and medium can give rise to a whole new set of process choices. For example, the decision to use a multimedia web site introduces a series of process choices related to maintaining the site that are not relevant in a static document such as a book. If the information is highly dynamic (e.g. sports scores), then design of the information product must include a means of facilitating the constant site updates.
Process choices are extremely complex and highly situational. Even the design of web sites is subject to situational analysis of audience, purpose, and environment. For example, the organization of a site intended to represent a company to the public has extremely different requirements than a site intended to facilitate the brainstorming of a working group. Because of this high degree of situational complexity, technical communication focuses far more on understanding the relevant questions and parameters that inform a communication process choice than it does on providing all-purpose answers to communication questions.
The effort by technical communicators to formally evaluate the various outcome variables discussed above is relatively new. For many in our field (though increasingly fewer), evaluation was seen as so complex and including so many variables that the best solution was to rely on expert judgment-themselves. With maturity, however, the field has come to realize that judgments like these must be demonstrated more objectively, though we still "lack a consistently used, accepted methodology for assessing the quality and value of our work." (Carliner 1997, p. 253)
Probably the most mature methodologies at our disposal are a set of techniques developed to evaluate user satisfaction and performance that have been organized under the title of "usability testing" (a.k.a. usability evaluation, usability engineering). Usability testing represents yet another rejection of the notion that general guidelines of quality can be developed that will apply in every communication situation. Instead usability testing is highly situational, meaning that the questions to be answered and methods for answering them change depending upon variables related to the nature of the communication product and when in the development process the testing occurs (Ramey ed. 1989).
For example, early in the development process, efforts to understand users and their environment can be ethnographic, almost anthropological in nature (Ramey, Rowberg, & Robinson, 1996). During development, specific questions can be addressed in "usability laboratories" (usually based on one-way mirrors and videotaping) that attempt to place the user in as natural a setting as possible and derive data to answer those questions from simulated performance, thinking aloud protocols, etc. Late in the process (e.g. beta testing), efforts are usually made to get out of the laboratory and view the user performing real tasks in their actual work environment.
Other methods for evaluating user issues include focus groups and surveys. Though extremely common and capable of providing important information, these are often unsatisfactory since there is generally a gap between what people will say about a product and how they actually use it. Another complicating factor is that user performance and satisfaction is not a static thing. Over time and with experience, what users care about and their reactions to a technology or an information product change. So, for example, a recent evaluation of an in-vehicle emergency communication device found that, initially, potential customers were concerned primarily about cost, but with use, they came to be far more sensitive to the bundled services that accompanied the communication capabilities (Haselkorn et al. 1997) For this reason, user satisfaction surveys must be repeated over time.
Evaluating customer or client satisfaction can be even more elusive. Carliner (1997) suggests that the best way to determine client satisfaction is to see if they give you repeat business or recommend you to others. Satisfaction surveys are generally less effective with organizational clients than they are for users because of the close working relationship between technical communicator and client, as well as the small sample size. Performance reviews, where both communicator and client together review objectives and how well they were met, can be more effective.
Thanks to the kinds of outputs identified by Redish and Ramey (Table 1), evaluating customer performance can be more straightforward. These outputs can be defined using numerical data that represent such things as reduced expenses, increased revenues, reduced hours, etc. However, these data can still be very difficult to agree upon and generate because it is necessary to isolate the contribution attributable to the efforts of the technical communicators.
Since business performance is a dynamic concept, any measure must be tracked over time before the communication project has an impact. Only then can we confidently assert that introducing the communication product or process caused the change. This means that the client and communicators must identify and agree upon measures of performance at the very outset of the project, and that data must be gathered prior to the project's completion.
Given the still relatively young efforts by technical communication to evaluate outcomes, it is not surprising that there are few if any formal models that have been applied to this effort. Technical communication's early focus on content, however, did lead to the application of models from composition and rhetoric that focused on production of text.
The only model worth mentioning here is the cognitive process model proposed and developed by Linda Flowers and John Hayes (Flower & Hayes 1981). The most significant impact of this model was a major shift in focus from the information product to the process by which that product was generated. This led to a further focus on particular sub-components of the process, such as planning or revising. Most relevant to technical communication were efforts to describe how adult writers planned or revised particular expository tasks such as writing proposals (Flower et al. 1989). Process theories like these were based on detailed analyses of actual writers and usually suggested goals for instruction and ways to support the activities occuring during particular process components.
Nevertheless, while these cognitive models have affected how we think about the communication process, they have not been applied to evaluation efforts, perhaps because the models themselves are incomplete or not fully specified.
There are two key messages from the field of technical communication concerning the evaluation of communication projects: (1) it must be carefully grounded in the specific communication situation and (2) it must be closely linked to the entire project process.
The first of these messages means there must be careful analysis of numerous input variables related to audience (users), purpose, and communication environment, and that subsequent evaluation should consider how well the purpose was achieved for that particular audience within the context of the actual communication environment. The transfer of results from one project to other projects will depend on similarities and difference in these three areas.
The second message means that measures of evaluation must be a part of project planning, and that these measures must be considered and evaluated throughout the development process. Given the complexities of a communication project, it is highly unlikely that evaluation measures can be successful fulfilled if the project personnel were unaware of them until after the activity.
Bandes, H. 1986. "Defining and controlling documentation quality."
Technical communication 33:6-9.
Carliner, S. 1997. "Demonstrating effectiveness and value: a process for evaluating technical communication products and services." Technical communication 42:26-39.
Carrol, J. ????
Coney, M. 1987. "Contemporary views of audience: a rhetorical perspective." The technical writing teacher. 14:319-335.
Coney, M. 1992. "Technical readers and their rhetorical roles." IEEE transactions on professional communication. 35:58-63.
Dumas, J. & J. Redish. 1994. A practical guide to usability testing.
Norwood, N.J. Ablex Publishing.
Farkas, D. ????
Flower, L. S., and J. R. Hayes. 1981. " A cognitive process theory of
writing" . College Composition and Communication : 32:365-387
Flower, L. S. K. A. Schriver, L. Carey, C. Haas, and J. R. Hayes. 1989.
Planning in Writing: The Cognition of a Constructive Process, Carnegie
Mellon Planning Project, 55 pages
Fran Buhler, M. ????
Haselkorn, M. 1989. "From on-line documentation to intuitive interfaces:
technical communicators join the design team," Journal of Technical Writing and Communication, Vol. 19, No. 4, 356-370.
Haselkorn et al. 1997
Horton, W. ????
Mathes & Stevenson ????
Nielsen, J. 1994. Usability Engineering. Boston MA. AP Professional.
Ramey, J., A. Rowberg, & Robinson, C. 1996. "Adaptation of an ethnographic method for investigation of the task domain in diagnostic radiology." in Field Methods Casebook for Software Design. John Wiley & Sons.
Ramey, J. ed. 1989. IEEE Transactions on Professional Communication special issue on Usability Testing. December.
Redish, J. & J. Ramey. 1995. "Adding value as a professional technical communicator." Technical communication 44:252-265.
Spyridakis, J. ????
Williams, T. & D. Farkas, "Minimalism reconsidered: should we design documentation for exploratory learning?" SIGCHI Bulletin 24(2), 41-50.
Williams, T. ????