Research Article

A Tool for Mapping Research Skills in Undergraduate Curricula

Gillian A Fraser, Anne C Crook and Julian R Park

Centre for Excellence in Teaching and Learning in Applied Undergraduate Research Skills (CETL-AURS), University of Reading

Date received:05/10/2006        Date accepted:08/01/2007

There has been considerable interest recently in the teaching of skills to undergraduate students. However, existing methods for collating data on how much, where and when students are taught and assessed skills have often been shown to be time-consuming and ineffective. Here, we outline an electronic research skills audit tool that has been developed to map both transferable and discipline-specific skills teaching and assessment within individual modules, the results of which can be collated and analysed across entire degree programmes.  The design and use of the audit tool is described in detail and a bioscience case study is presented to illustrate the types of data that can be collected. The audit tool has been designed as a time-effective way of collecting information on skills teaching and assessment, but also actively encourages staff to reflect on their teaching and learning practices. Conclusions are drawn about the practicalities of using the audit tool and its importance in both curriculum design and as a resource to encourage dialogue with graduate employers.

Introduction

One of the interdisciplinary deliverables from the University of Reading's Centre for Excellence in Teaching and Learning in Applied Undergraduate Research Skills (CETL-AURS)[1] was to map the teaching and assessment of research skills.  For the purpose of this study, research skills were classed as a combination of transferable and discipline-specific skills. 

Increasingly, employers are demanding that graduates enter the workplace with the necessary skills and knowledge to benefit the organisation with little further training, particularly with regard to transferable skills.  These essential and desirable skills include communication (written and oral), IT and team-working (Owen, 2001).  Many academics will insist that the use of transferable skills is intrinsically embedded within curricula to a greater or lesser extent; for example, course work is expected to be submitted before deadlines, oral communication skills are used extensively during in-class presentations and collecting additional information is essential for research projects, exam revision etc.  These few examples are dependent on time-management, IT, communication and information handling skills.  The question is, however, are students taught how to use and further develop transferable skills, and where does this happen within the curriculum?

Several universities have perceived the teaching of transferable skills to be important enough to warrant the creation of compulsory skills modules.  For example, psychology students at London Metropolitan University must attend a discipline-based skills module (Elander, 2003).  The Universities of Exeter, Middlesex, Dundee and Gloucester also offer specific skills modules.  The use of dedicated modules to teach transferable skills has however been questioned. Knight and York (2002) suggested that 'skills learning' is too complex a process to be satisfied by a single module; they argue that embedding the teaching and practice of transferable skills throughout a degree programme allows the student to utilise and develop skills in different situations, which is more effective than using the skill under an artificial situation. 

By their very nature, discipline-specific skills are embedded across a range of modules.  The degree to which these skills are reviewed, in terms of how applicable or appropriate they are, both in terms of the curriculum and to employers, is usually dependant on individual module co-ordinators.  Within the sciences, as technology improves and experimentation results in a greater understanding of processes, the type and range of discipline-specific skills being taught is likely to change, which is important because the needs of employers are also likely to change over time.  The Higher Education Academy has recently produced a range of 'Student Employability Profiles', which list the skills that should be gained by students studying particular disciplines.  It is anticipated that, in response to these profiles, HEIs will be under increasing pressure to identify where and how discipline-specific skills are being taught and assessed within curricula.  Regardless of the method of delivery, there is increasing pressure to identify skills learning opportunities in degree programmes and in individual modules (deVries and Downie, 1999).  It has been suggested that when developing a new programme, the skills criteria can be identified at the initial planning stages, therefore modules can be tailored to teach and assess the desired skills (Borthwick, 2002).  For existing programmes, however, identification of existing skills teaching and assessment is required to determine whether or not there are any deficits in skills learning opportunities.  

Although skills mapping within curricula per se is not a novel idea, previous work in this area has relied on a range of complicated processes that are often time consuming to implement and analyse.  To date, there have been three main approaches to skills mapping:

1. Interview-led discussions with either the course director (Mistry et al., 2006) or individual module convenors (Kemp and Seagrave, 1995).  This can be time-consuming and course directors may not always be able to identify precisely where in the degree programme certain skills are taught (Knight and Yorke, 2000).  In addition, time constraints render this method inappropriate for programmes where students have a high degree of module choices beyond their core options. 
2. Longer-style open questions have been used to provide intensive qualitative information regarding skills teaching.  However, the quality of responses may vary depending on the willingness of the participant, and the lack of uniformity of the answers may present difficulties during analysis (Kemp and Seagrave, 1995).
3. Matrices to indicate what skills are taught and/or practised and/or assessed within individual modules (Chance, 1995b).  This method is often a 'tick box' based approach and details the skills taught within individual modules.  Analysis of results may prove problematic depending on how the matrix is compiled.

Although it is not uncommon to audit either a range of transferable skills (Chance, 1995a,b), or to concentrate on one specific skill (Vickery and Cooper, 2003), few studies have attempted to map what we describe here as 'research skills', i.e. transferable skills and discipline-specific skills.  Here we describe an on-line audit tool, which has been designed to enable module coordinators to provide quick but detailed information on the range of research skills being taught and assessed within their modules. Data can then be collated across modules comprising a degree programme to provide an in­depth analysis of research skills teaching and assessment across the entire curriculum.

The CETL-AURS Research Skills Audit Tool

The research skills audit tool is one of a number of approaches CETL-AURS has adopted to investigate research skills teaching, learning and assessment.  Parallel projects include the development of resources for prospective students to describe the importance of research within undergraduate curricula, and a series of student questionnaires and linked workshops, which attempt to quantify students' perceptions and competencies in research skills.

The aim of our research skills audit tool is to provide a quick, user-friendly method to identify the range of research skills being taught and assessed within individual modules, as well as providing information on the types of feedback provided to support students' future skills development.  Modular data can then be combined to provide degree programme-level review information.  This is an important step because whilst it is clear that research skills teaching is firmly embedded within curricula at the University of Reading and elsewhere, detailed information regarding where or how these skills are taught is often lacking, which in the past has brought into question the scope for skills progression within curricula.  One key difference between this tool and similar work that has been done in the past is that in this case the auditing of discipline-specific skills will be used for comparison with the HEA's recent Student Skills Profiles data, which in turn will serve as a  basis for staff to discuss graduate skills  with employers.  Our aim is that this will allow the teaching of discipline-specific skills to be better aligned with industry needs and expectations. 

To support the development of the research skills audit tool specialist software called PROFILE (www.profile.ac.uk), developed at the University of the West of England, was used to aid the tool's interactivity, data collation and ultimately the wider dissemination of this skills mapping resource.  For the purpose of the research skills audit tool, skills were defined as those that either constitute a Reading Graduate transferable skill[2] or a discipline-specific skill e.g. microscopy, surveying, invertebrate field sampling etc. 

Overall the research skills audit tool comprises four discrete sections and to make this resource as user-friendly as possible answers to questions generally require no more than a box to be 'ticked' or left unchecked, together with an option to provide more detailed information.  The audit starts with the user entering details of the module, including its title, module code, credit weighting and the degree programme within which it operates.   

Figure 1  Section 1 of the research skills audit tool

The first section of the audit tool (Figure 1) investigates the types of transferable and discipline-specific skills being taught and the range of teaching methods being employed to teach these skills.  A free text box allows the user to enter as little or as much information about the discipline-specific skills.

Section 2 requires information to be entered about the methods of assessment for each of the transferable and discipline-specific skills, and whether or not these assessments are formative (for feedback purposes only), summative (contribute to the module marks) or a combination of the two (Figure 2). This section makes use of drop-down boxes to facilitate ease and speed of data entry.

 Figure 2  Section 2 of the research skills audit tool

The third section of the audit tool asks for information about the feedback given to students and whether or not it is oral or written, formative, summative or both. The fourth and final section asks the user three 'trigger' questions about the information they provide to students within module description forms (Figure 3).  In a similar manner to the Centre for Bioscience's assessment audit tool, at the end of the research skills audit there is a text box entitled 'Action Plan', which has been provided to encourage users to reflect on the responses they have entered.

Module co-ordinators are required to 'log in' to the Profile system, which ensures the results of their individual audits remain confidential and can only be accessed by other staff by invitation.  The exception to this is the programme director ('Administrator') who has automatic access to the relevant module data within his/her degree programme.  Once the module coordinators have completed their audits, the programme director can immediately produce a report of the results directly from within Profile. The resultant data table can then be imported into Excel, allowing for quick and easy data manipulation, analysis and presentation.  These data can then be used by the programme director to evaluate skills provision within and between modules across the curriculum. In addition, the generation of detailed lists of research skills teaching and assessment provides a useful starting point for discussions with employers, and in particular to facilitate a comparison of the graduate skills expectations of employers with what is actually currently provided within curricula.

Figure 3  Sections 3 and 4 of the research skills audit tool

Evaluation of the Research Skills Audit Tool

Module coordinators were asked to comment on the audit tool, with particular emphasis on the appropriateness of the questions and on its user-friendliness.  Several staff commented on the layout of the forms (smaller screens require the user to scroll horizontally to see the whole form) and in some cases the uncertainty of what actually constituted a 'discipline-specific' skill.  On average, module coordinators took 16 minutes to log in and complete a form for a single module.  Of the evaluation forms returned, 83% of staff either agreed or strongly agreed that the audit tool was user friendly; over 80% agreed or strongly agreed that the audit helped them reflect on their teaching and assessment methods and the information provided in the Module Description Forms[3] (MDF), and that they would implement changes to their MDF as a direct result of completing the audit tool.  In addition, several module coordinators indicated that the audit had led to them review the methods of assessment they were using and had raised questions regarding an apparent lack of formative assessment within their modules. We therefore believe that the research skills audit tool has been a success both in terms of its ability to quickly generate detailed skills information within and across modules but also in terms of its use as a resource to support reflection on skills teaching and assessment.

Case Study: BSc. Rural Environmental Sciences (RES)

The RES degree programme combines the study of biological, environmental and landscape sciences and aims to provide practical and scientific understanding of the processes associated with rural resources. RES graduates enter a wide range of careers, including environmental consulting, government agencies, wildlife organisations etc.  Students study for a total of 120 credits each year, half of which are related to core or compulsory modules with the remaining 60 credits being chosen from a list of 15 elective modules.  The research skills audit was completed by 11 module co-ordinators who were responsible for the 13 core (compulsory) modules[4]. Here, outcomes from the audit are presented for selected modules and for the programme as a whole to give an indication of the range of data that can be collected.  A sample of the results from first and third year core modules is illustrated in Table 1. 

'Introduction to RES' is a first year module that forms the basis for the field course module at the end of the first year as well as several related modules in the second year.  The results of the audit indicate that this first year module teaches a range of discipline-specific skills that go on to form the basis for subsequent modules to build upon.  A total of six different teaching methods are used in this module using techniques that encourage students to work within a team, to introduce their own thoughts and ideas to the subject and to gain a range of practical skills.  The students are assessed using a range of techniques that provide effective reasonable alignment with the module's stated learning objectives.  After completing the audit, the module coordinator reflected on the teaching and assessment of the skills and decided that it would be beneficial to make skills teaching more explicit, i.e. make sure students were aware of both the transferable and discipline specific skills they were learning and developing. A further area identified for improvement in this module was the need to diversify the methods of assessment being used (although several different pieces of assessment are completed over the year, the majority are writing based).

Table 1 A sample of  results from the research skills audit taken from two core modules within the RES degree programme

Module title	Transferable skills taught	Discipline-specific skills taught	Teaching methods used	Assessment methods used	Action Plan
Introduction to Rural Environmental Sciences (20 credit Part 1 module)	Written communication Oral communication IT Numeracy Information handling	Ecological sampling Taxonomic skills Experimental design Statistical analyses	Small group teaching Practical work Field classes Discussions IT practical work Tutorials	Field report Field assessment Oral presentation Project Literature Review Critique	Modify MDF to include skills matrix Assess all specialist skills Avoid repetition of assessment method
Geographic Information Systems and Simulation Modelling (10 credit Part 3 module)	Written communication Problem solving IT Numeracy Information Handling	Simulation modelling Systems thinking Spatial analysis using GIS	Lectures Small group teaching Practical work One to one tutorials Student group work	Laboratory report Project Literature review Calculations	Introduce pre-requisites Modify MDF to include skills matrix Use formative assessment early in module

'Geographic Information Systems and Simulation Modelling' is a third year module, which introduces several discipline-specific skills that students have not previously been exposed to.  As for 'Introduction to RES' the module co-ordinator in this case also felt that skills learning opportunities could be better articulated to students. The module co-ordinator also concluded that there was a need for enhanced formative assessment opportunities early on in the module; this was felt to be particularly important because the students had not previously been exposed to some of the discipline-specific skills being taught.

The data from the core modules were collated to provide information about the central components of the entire degree programme.  Overall transferable skills relating to the 'Reading University Graduate' appeared to be well represented in the curriculum; oral communication was the least represented skill, but was still taught in 50% of the core modules. 

At the degree programme level it is not surprising that lectures and seminars were the most popular method used to teach both transferable and discipline-specific skills although the programme does contain a high level of field and laboratory work. Several module coordinators used different methods, including debates, oral presentations, posters and role play, to support both transferable and discipline-specific skills learning and development.  The output from the audit illustrates that across the core modules, 13 different methods are currently being used to teach transferable skills compared to nine methods for the discipline-specific skills. Table 2 provides a list of discipline-specific skills being taught and their methods of assessment.

Table 2 Discipline-specific skills and methods of assessment for core modules within the RES degree. (F = formative assessment, S = summative, B = both)

Module	Discipline-specific skills taught	Methods of Assessment
Introduction to RES	Ecological sampling Taxonomic skills Experimental design Statistical analyses	Field report (B) Field assessment (B) Oral presentation (F) Project (B) Critique (F)
Biodiversity	Ecological field survey Microscopy	Field Report (S)
Physical Ecology	Interpretation of climatic data Environmental instrumentation Critical reasoning	Calculations (B) Exam (B)
Soils, Land and the Environment	Anion/cation exchange Use of specialist software Identification of soil properties	Virtual Field Report (S) Exam (S)
Ecological Aspects of Environmental Assessment	Interpretation of Environmental Policy	Coursework (S)
RES Field Course I	Experimentation Taxonomy Scientific writing Sampling Data Analysis General scientific skills	Lab Report Field Report Field Work Observation
Practical RES	Using keys Field Sampling Techniques Microscopy Scientific paper reading	Oral presentation (B)
Statistics for Life Sciences	Data Analysis Use of Minitab	Assessed exercises (F) Exam (S)
Sustainable Land Management
Approaches to Sustainable Development	Life cycle analysis Participatory methods Multi-criteria decision making	Project (S) Oral presentation (S) Debate (S)
Geographic Information Systems and Simulation Modelling	Simulation modelling Systems thinking Spatial analysis using GIS	Laboratory report (B) Literature review (S) Project (B)

It was noted in several modules that discipline-specific skill were taught but not assessed.  In particular, the Practical RES module coordinator highlighted this and is amending the module to include more formative and summative assessment of discipline-specific skills.   The research skills audit has highlighted the need to ensure progression in the teaching (and assessment) of data analysis and taxonomic skills throughout the programme. To this end the programme director is now encouraging the use of more project work within the core field work modules with the aim of further embedding the teaching of these skills.  Another key area for improvement arising from the audit is in relation to career management skills and the results of the audit have fed into a School-wide programme to raise student awareness of career management skills.  This is now being achieved through the use of industry visits, talks from employers and a new initiative to encourage students to engage with an industrial training year between their second and third years.

Discussion

The aim of the research skills audit tool was to create a user-friendly method to map where and how research skills are being taught and assessed within individual modules and across entire degree programmes.  Given the large quantity of paperwork academics are expected to complete on an almost daily basis, and the voluntary nature of the audit tool, it was essential to minimise the time taken to complete the form, whilst at the same time, maximise the quality and range of data collected.  Most module coordinators required between 15 and 20 minutes to complete an audit of their module, and on reflection, the majority found the experience to be useful.  Although previous research has given no indication of the time taken to complete similar audits, it is envisaged that cross-referencing the skills listed on module description forms with university standards (deVries and Downie, 1999) or open questions (Kemp and Seagraves, 1995) would take considerably longer to achieve.  In addition, with a basic knowledge of Excel, manipulation of module data to generate an overview of an entire degree programme's skills 'profile' (conducted by the programme director), takes on average less than an hour.  

On an individual module basis the audit has led to staff thinking critically about how skills are taught and assessed within their modules.  This has already resulted in many participants suggesting amendments/improvements to their teaching and assessment.  For instance the Field Course relied heavily on written assessment methods, but the module coordinator is now considering other more practical methods in the field.  In addition, many of the respondents identified the need for more formative assessment within their modules; some staff who took part in the audit have also made amendments to their module description forms to better articulate skills learning opportunities to students.  

From the perspective of the programme director the collation of data from core modules provides a useful starting point, which we envisage could be useful in a number of scenarios, including:

• As a basis for reviewing skills provision within a degree programme to ensure the necessary skills are being taught and developed, and opportunities for skills progression are present throughout the curriculum.
• As a basis for discussion with employer groups or professional bodies to compare skills expectations and in particular to ensure that students are being equipped with the opportunities to learn and develop the necessary skills required by industry.
• To support discussions with students to articulate the range of skills learning opportunities within individual modules and throughout the degree.
• To provide data for Periodic Review or for other quality assurance purposes.

Conclusion

The research skills audit tool has been well received by both module coordinators and programme directors.  On an individual module basis, the audit has provided a systematic method for staff to reflect upon their teaching and assessment of research skills.  We have developed a resource that is quick and easy to use and which can provide quality data for both modular and programme-level skills review.  One of the main advantages of the audit tool is that the data can be easily collated and used for a range of purposes, including Periodic Review, development of new modules, revising existing modules, student focus groups and employer discussions. The audit tool is available for use at other institutions, and people requiring further information should contact Dr Gillian Fraser for further details, g.a.fraser@rdg.ac.uk.

Acknowledgements

The authors would like to thank the range of staff who have taken part in the development and use of the Research Skills Audit tool. We would also like to thank Dr Stephen Gomez at the University of the West of England for his help and support in relation to the use of PROFILE system.

Communicating Author

Gillian Fraser, Centre for Excellence in Teaching and Learning in Applied Undergraduate Research Skills, School of Agriculture, Policy and Development, University of Reading, Earley Gate Campus, Reading, Berkshire, RG6 6AS. Phone 0118 3786786, Fax 01189352421. Email g.a.fraser@rdg.ac.uk

References

Borthwick, F.  (2002).  Skills Plus:  Audit of the undergraduate higher national programmes in the construction group at Liverpool John Moores University.  Available at:  http://www.cebe.heacademy.ac.uk/learning/casestudies/case_pdf/Borthwickf.pdf#
search=%22skills%2C%20curriculum%2C%20audit%2C%20undergraduate%22  (accessed 13 September 2006)

Chance, J.  (1995a).  Skill-based profiling at Oxford Brookes University.  In A. Jenkins & A. Ward (Eds.) Developing Skill-based Curricula Through the Disciplines: case studies of good practice in Geography, SEDA Paper 89.  Available at http://www2.glos.ac.uk/gdn/seda/chance.htm (accessed 13 September 2006)

Chance, J.  (1995b).  Oxford Brookes University: An integrated curriculum.  In A. Jenkins & A. Ward (Eds.) Developing Skill-based Curricula Through the Disciplines: case studies of good practice in Geography, SEDA Paper 89. Available at http://www2.glos.ac.uk/gdn/abstracts/a114.htm (accessed 13 September 2006)

Elander, J.  (2003).  A discipline-based undergraduate skills module.  Psychology Learning and Teaching, 3 (1), 48-55

Kemp, I.J. and Seagrave, L.  (1995).  Transferable Skills - can higher education deliver?  Studies in Higher Education, 20 (3), 315-328

Knight, P. and Yorke, M.  (2000).  Skills Plus:  Tuning the undergraduate curriculum.  Available at http://www.heacademy.ac.uk/documents/paper1.rtf (accesssed 13 September 2006)

Mistry, J., White, F. and Berardi, A.  (2006).  Skills at Maters' level in geography higher education: teaching, learning and applying.  Planet, 16, 9-14

Vickery, S. and Cooper, H.  (2003).  Confidence of competence? - auditing information literacy skills of biology undergraduate students.  Available at http://www.lib.mq.edu.au/about/conferences/educause/competence.pdf#
search=%22skills%2C%20curriculum%2C%20audit%2C%20undergraduate%22 (accessed 13 September 2006)

deVries, P. and Downie, N.  (1999).  Ensuring skills' breadth across the curriculum: a case study and a method.  UltiBase Articles, available at http//:ultibase.rmit.edu.au/Articles/dec99/devries2.htm (accessed 12 September 2006)