Evaluating the relationship between government agencies and their website audiences
In evaluating government webpages we have used the Quality of Relationship (QoR) framework to guide the measure of website usability. The QoR is a powerful, cost-effective way to find out if your website is meeting its goals. You can use QoR to find out if your different audiences are getting what they need from your website. It has been shown to be effective for large federal websites, and it provides information that can be used to develop the site, improve usability, and gain visibility and trust.
The QoR metric has been developed for assessing relationships between agencies and their audiences. It is a GRPA[i]-approved metric for evaluating performance of government websites.
The Quality of Relationship is based upon five dimensions:
- Awareness—A public knows the agency exists, what it does, and why.
- Trust—A public perceives the contents of the agency’s products/services to be accurate, credible and authoritative.
- Satisfaction—A public perceives the agency’s products/services to be relevant, reliable, and complete.
- Usability—A public regularly uses the agency’s products/services, and considers them to be easy to use.
- Control Mutuality—An audience perceives an ability to interact with and influence the agency, and vice versa.
Data are collected from an agency’s audiences to discover the specific characteristics of each dimension for each audience segment. Methods are used to both cast wide nets and drill down to gather in-depth perceptions from the public. Quality of Relationship is adapted for non-commerce organizations from the American Customer Satisfaction Index, which is an economic indicator that measures the satisfaction of consumers across the U.S. economy (http://www.theacsi.org/). Methods and characteristics for evaluating the relationships between the public and organizations were described by James Grunig at the University of Maryland in 2002 (http://www.instituteforpr.org/wp-content/uploads/2002_AssessingRelations...) and have been enhanced for QoR analyses.
Data collection methods may include any or all of the following:
Web Analytics—Analytics software such as Google Analytics are easy to install, collect, and analyze. They provide comprehensive data about website visitor behavior, demographics, and visitation patterns.
Fly-in Surveys—A fly-in survey can be offered to website visitors who have stayed on the site for a specific amount of time (for example, 30 seconds). If the survey is fairly short and simple to use, this is an excellent way to collect feedback from a wide variety of users.
Usability Studies—Members of particular audiences are recruited to participate in an interactive session that allows them to explore the website to obtain real-time feedback on the visitor’s perspective. Activities are aligned with the goals of the website designers. Participant feedback and behavior during the session are analyzed to provide a rich set of detailed data.
The QoR analysis provides a numerical assessment for each area of the agency/audience relationship. A sample figure presenting such data is seen below.
Phases of Study
Phase I: A Quality of Relationship study starts with understanding the goals of the organization, interviews with the project team and leads, and development of research questions.
Phase II: A smaller scope study should include the web statistics analysis and the fly-in survey, from which the Quality of Relationship metric is derived.
The web analytics for the whole study period is compared to those during the survey period to be sure the samples are comparable.
Phase III: If a more in-depth study is desired, usability protocols are developed that reflect desired important tasks users may do on the site. An interest survey is implemented to be able to select usability participants that reflect the audience as defined within the fly-in survey. Usability study interviews are conducted virtually, which provides the ability to reach power-users, users across geographic location, and users who would otherwise choose not to participate.
Phase IV: Final reporting synthesizes findings from across the methods. A webinar is provided for the project team, with follow up conversations to help the team understand the findings and decide how to implement recommendations in their work.
[i] Government Performance and Reporting Act FAQ http://www.performance.gov/faq#Q4