Kate Silton, North Carolina A&T State University
May Yan, Ryerson University
Assessment of their institutions’ implementations of Summon
Assessment in general:
- What are your goals? Who are your stakeholders?
- Scope should reflect resources available to carry out project
- Assessment projects should be specific and targeted
North Carolina A&T: Assessing Summon’s freshman year
- How frequently does it deliver the full text of the article?
- Will students stop using other databases?
- Who is using this tool and how are they using it?
1. Delivery of full text
- Did a search, tested first 100 results — did it work?
- About 60% got to full text (originally it used link resolver, after Summon implemented direct links it got better)
- Tested out some limiters as well – newspaper full text hard to get to, but sometimes the metadata for these are bad and hard for link resolvers to work with
- ebooks were difficult as well as first, got better with direct linking
- delivery of full text by vendor: some vendors have worked better than others – major problems with Gale links until direct linking
2. Impact on e-resources usage
- Proquest usage bumped up, Springer too, others look like they went the other way
3. Who’s using Summon?
- admin console provides lots of data about users
- most visits from IE9 on Windows
- surprising number of users on dialup
- lots of users using it on campus, and via proxy server
- top queries: electrical engineering, blank search, usual first year Qs
- systematic plan to deal with broken links
- continue to monitor usage
- follow up usage analytics with a survey or focus groups
Summon evaluation at Ryerson University
- goals: evaluate investment in Summon, determine where to devote resources to maximize investment, increase awareness of Summon
- methodology: questionnaire (whole university), focus groups (students) and interviews (faculty/researchers)
- wanted study to be repeatable over the years
- How is Summon being used and who is using it?
- are they satisfied with it? Rate ease of use? any usability issues?
- Has there been enough promotion of Summon? Info on how to use it? What other info needs to be provided?
- Contrast experience between user groups: undergrads/grads & faculty, novice vs. experience library users, disciplines of study, Summon and other projects (Google Scholar, subject indexes)
- online survey using Survey Monkey
- 10-15 questions
- open and close ended questions
- all faculty, staff and students
- advertised on library homepage, posters, LCD screens, mailing list to all students and faculty (in phase 2 only)
- incentive: bookstore GCs in first phase, 3 iPads in second phase
- Inductive process relies on getting ‘good’ feedback from open ended Qs in questionnaire and focus group/interview — more challenging to carry out consistently over time
- two questionnaires because the first was intended as a learning experience, so they’d know how to do it best in phase 2 (marketing too)
Focus groups and interviews currently happening
Response rates & results:
- phase 1, 191 responses, over half had used it and 75% said they’d use it again
- phase 2, 6344 responses – direct email to mailing list worked best
- for last academic search, 60% had used Summon – mostly for writing essays, also used web search engines, library website, professors and instructors, other websites (not search engines)
- How easy was it to use? Mostly very easy
- Easy to find what they needed? Moderately easy category was the highest
- How satisfied? Very-moderately satisfied
- Many evaluation strategies, might want to use combinations to get a more complete view
- expect that it will take time for a shift in user behavior and plan evaluation accordingly