Abstract
The choice between the subject and the object approaches and their combination is one of the challenges designers of innovation surveys face. Although it is encouraged by the Oslo manual, the combined use of these two approaches is rare. The paper presents a large scale education sector innovation survey using these approaches simultaneously in a matched employer/employee data collection with a special focus on small innovations initiated at grassroots level.
Measuring innovation in the education sector is a major item in the portfolio of education-related activities of the Organisation for Economic Cooperation and Development (OECD). In the past decade this organisation published three books on measuring innovation in education (OECD, 2014; Vincent-Lancrin, Jacotin, Urgel, Kar, & González-Sancho, 2017; Vincent-Lancrin, Joaquin, Soumyajit, & Gwénaël, 2019). This work has been conducted with the direct financial support of the European Commission, demonstrating the recognition of the political importance of this activity. This educational innovation related activity of the OECD is closely related with the implementation of the general innovation strategy of the organisation (OECD, 2015).
Understanding the problems of measuring innovation in education requires understanding the measurement of innovation, in general, and in the public sector, in particular. Among the challenges innovation surveys face four deserve special attention: (1) the object/subject approach, (2) the scope of innovation, (3) the definition of the subject (innovating agent) and (4) the inclusion of the diffusion dimension. A further challenge is related with the specificities of the education sector, and with the particular approaches to innovation mechanisms in this sector. These challenges raise also a number of methodological issues, such as, how to cope with multidimensionality and how to define the units of analysis.
In this paper we present some of the outcomes of a large scale education sector innovation survey with a special focus on the challenges mentioned in the previous paragraph. In the first part we present the general theoretical context providing a brief analysis of each of the challenges mentioned. In the second section we provide a short presentation of the survey. The third section provides a brief description of the specific national context of the conception and the implementation of the survey. In the fourth section a number of relevant outcomes are presented to illustrate the analytical opportunities opened by the combined subject/object approach and by the use of matched employer/employee data. Finally, the last part is devoted to concluding remarks.
Theoretical and methodological context
In his book about the history of measuring scientific and technological progress Godin (2004) evokes the debates accompanying the first innovation surveys and mentions “choosing the approach” as one of the key dilemmas. Since innovation can be seen both as an outcome or product and as an activity or process, one of the major questions has been to choose between what the Oslo manual calls “object approach” and “subject approach” (OECD/Eurostat, 2018). In the case of the first, specific innovations (products) are identified and those creating or adapting them are asked about this specific innovation. As for the second, the unit of analysis is not the innovation itself but the productive units (firms or other actors), and their general innovation behaviour or activity is analysed, in a way disconnected from their specific innovations.
The last edition of the Oslo manual devoted a whole chapter to the use of the object approach with a special focus on “combining” the two approaches. This combination can be realized through completing a subject approach questionnaire with an object approach module, the latter focusing on one specific “focal” or “most important” innovation, selected by the interviewed agent. The manual mentions a few specific innovation surveys where this combined approach has been applied. This solution might result in data on a high number of different and specific innovations, that is, its outcome is a dataset in which specific innovations can be taken as cases and each of them might have a number of quantifiable scalable attributes. According to the Oslo manual there are three main benefits of including questions about unit-specific innovations in innovation surveys: it allows to cover (1) a large number of innovations, (2) all kinds of innovations and (3) innovations new only to the observed organisation (OECD/Eurostat, 2018).
Although the Oslo manual gives the same labels to them, it is important to note that the classical object approach is fundamentally different from the one that is based on the identification of one “focal” or “most important” innovation selected by the surveyed subjects. When the combined approach is used the number of innovations (objects) observed might be as many as the number of surveyed subjects and they typically represent a high level of variety. This is fundamentally different from the classical object approach where several subjects provide data on the same innovation (the use of digital learning management in the education sector is a good example).
As Arundel, Bloch, and Ferguson (2019) points to it “the inclusion of questions on a single innovation has substantial advantages: it permits questions that may be too difficult for respondents to answer for all of their innovations combined” (p. 8). The use of both the subject and the object approaches in the same survey instrument might be particularly useful, or even necessary in the case of “small innovations” which are predominant in the public services. In fact, a specific challenge faced by researchers of public sector innovation is how to deal with “small” innovations initiated by “frontline” workers when facing everyday problems in their professional activity. This has been analysed, for example, by Arundel and Huber (2013) in their paper trying to explain why public sector innovation surveys find surprisingly high-level innovation activity in public service organisations. Small innovations realized by frontline professionals in their daily practice may play an important role in improving the quality and effectiveness of public services. They can be described as “hidden innovation” (NESTA, 2007), “invisible innovation” (Fuglsang, 2010), “everyday innovation” (Lippke & Wegener, 2014), “employee-driven innovations” (Høyrup, 2012), “bricolage” (Bugge & Bloch, C. W. 2016; Fuglsang, 2010) or – specifically in the education sector – “school-based” (Kärkkäinen, 2012) or “teacher-led” (Georgsen, Beard, Ilomäki, Mor, & Koskinen, 2015) innovations.
Small innovations are particularly important also in the education sector. As the NESTA (2007) study on hidden innovation in five “low innovation” sectors noted: „in education the bulk of new practice is developed by individual schools and teachers”. Although in education systems, typically operating under strong national control, the most significant and most visible innovations are initiated by governments designing and implementing reforms, there are thousands of small, less visible innovations initiated by frontline practitioners or institutions. This has become especially important with the penetration of information technology, supported often by larger or smaller IT companies, cooperating with networks of voluntary institutions while initiating innovative solutions. Some of the small, local innovations can grow into a standard solution that spreads across the system, although most of them do not cross the threshold of being used only in a limited number of places or take various forms and names in an evolution-like process.
A further challenge innovation surveys might face is how to define the innovating agent. Most surveys follow the subject approach of the Oslo manual where the innovating agent is the firm or – in the case of public sector innovation surveys – a provider of public service or a government agency. The individual agent (employee level) rarely appears in large scale innovation surveys due to the technical difficulties of matching individual employees with specific workplaces and also to sampling challenges. There have been, however, some attempts to extend the analysis from the unit (firm, public service provider) to the individual (worker, employee) level through linking databases of innovation surveys and matched employer-employee data. One of the recent examples is the analysis by Dostie (2018) on the impact of the training of employees on the innovation activity of their workplace. In this case the researcher merged data from workplace and from innovation surveys, originally created for different purposes.
As the history of innovation surveys, and in particular the European Community Innovation Survey reveals (Arundel & Smith, 2013; Godin, 2004) one of the key issues designers of questionnaires have been facing was whether the diffusion dimension should be included, that is, innovations which are new to one specific agent but not necessary to the „world” or to the given sector should be covered. The approach of the Oslo manual includes “new to the firm” innovations which are typically generated by diffusion. This has opened the way to focus not only on the creation of new solutions but also on their adaptation. This is a natural implication of the shift to the subject approach which encourages the analysis of organisational features relevant in an innovation perspective, such as adaptive capacity.
Research on innovation in education has a long history. In fact, for specific reasons, the problem of innovation in this sector became the object of research earlier than in many other sectors (Arundel & Huber, 2013). Paradoxically research on educational innovation has not been connected for long to the mainstream of innovation studies, including those on public sector innovation. This is well-illustrated by the fact that those doing research in the highly institutionalized area of “educational change” almost never refer to publications in the field of general innovation studies. The International Handbook of Educational Change, for example, edited by leading specialist in this area (Hargreaves, Lieberman, Fullan, & Hopkins, 2014), does not make any reference to the Oslo manual which is one of the most important reference work for those doing innovation research.
In principle, the use of the classical object approach is possible also in the education sector. There are generally recognized inventories of major education innovations which could be the basis of surveys using the classical object approach. One example of this is the “Innovating Pedagogy” reports of the UK Open University, another, in the field of IT based innovations, is the “Horizon reports” published by a leading US agency. These inventories, however, cannot give an appropriate account of the rich world of small innovations and experimental solutions invented and used every day in schools and classrooms.
In this paper we present an illustration of how the theoretical dilemmas addressed above can be challenged simultaneously in an innovation survey, and of the analytical opportunities opened by this. We use as an illustration an education sector innovation survey implemented in Hungary in the framework of a research project which aimed at exploring local, grass-root level or bottom-up innovation processes throughout the whole education sector (The “Innova project” funded by the Hungarian National Research, Development and Innovation Office. Project number: 115857). It was initiated as the follow-up of another project on the classroom level impact of major centrally initiated education development programs which had led not only to the local adaptation of centrally initiated “big innovations” but also to the proliferation of “small innovations” (Fazekas, 2018).
The survey design of the Innova project can be described as an original way of (1) combining the subject and object method, (2) focusing on small (hidden, employee-driven) innovations; (3) using a matched employer-employee approach and (4) focusing both on innovation creation and innovation diffusion. This design has raised a number of methodological challenges which will be presented in the following section.
Some outcomes of the Innova survey have already been presented elsewhere (see Halasz 2018a, 2018b, 2020). In this paper the focus is on the analytical perspectives opened by the simultaneous use of different survey approaches and techniques following the perspective of theoretical considerations presented above. A central concept in this paper is the “value of innovations”. One of our objectives is to present a tool to evaluate the significance or value of locally initiated “small innovations” and to explore those factors that might influence this significance or value.
Methodological and survey design challenges
The survey instruments used in the Innova project were created on the basis of a general theoretical model of innovation in the education sector (see Halász, 2018a). Two rounds of large scale data collection were implemented (the first in 2016, the second in 2018) with electronic questionnaires (containing a number of common questions) sent to all education units (more than 10 thousand schools and university departments) in Hungary. This was supplemented (in 2019) by a third, small-scale pilot data collection in 100 units with a questionnaire used in face to face interviews. As we shall see later, this latter solution allowed us also to test of complex models, such as, the Michigan Innovation Research Project model (Van de Ven, Angle, & Poole, 2000) and our own theoretical model, on specific innovations by trained interviewers.
The respondents in the first data collection were leaders of the surveyed units, in the second and third (supplementary) data collections the respondents were both unit leaders and some of their employees (teachers). In the first data collection round we received responses from more than 4,800 units. In the second round responses arrived from nearly 2,000 units and more than 4,000 of their staff members. Most of the questions about innovation activity and behaviour invited respondents to report on the frequency of specific events in their practice (occurrences described as “never”, “a few times”, “often”, “very often”) and the word “innovation” was not used in the questionnaires (a frequent alternative was “new solutions”). The composition of the respondents does not represent the national education system, so the databases cannot be used to evaluate the general state of education innovation in this system. They offer, however, a unique opportunity to analyse the dynamic of innovation processes in the education sector.
A key challenge faced in the process of designing the data collection instrument was similar to what innovation researchers face in general when they are confronted with the question whether it is possible to create a common innovation definition that cover all sectors. As Gault (2018) noted: “the benefit of using a general definition of innovation is that innovation can be measured in a consistent way in all sectors and new indicators developed that describe the interactions between actors in sectors and between sectors.” This question was in the centre of debates accompanying the development of service sector innovation measurement, opposing the “assimilation” and “demarcation” approaches (Gallouj & Savona, 2010). In the case of the Innova project the survey questionnaire had to be designed so that it can be used in all sub-systems of the education sector, from kindergartens and elementary schools, through vocational training providers to university departments and doctoral schools. This required a thorough process of cognitive testing with the representatives of all sub-systems. One of the interesting difficulties emerged at the level of higher education: respondents here sometimes had difficulties to distinguish innovations generated by them in the social/economic environment of their institution and innovations realized within their own organisation.
All data collected during the three data collection rounds were merged into two databases: one on organisational units (employer level), and another one on individuals (employee level). The former also contains the aggregated data of employees working in the given organisation and the latter contains the data of the organisation where the given individual works. This allows a number of multidimensional and multilevel analyses. Since organisational data were collected in three rounds, for quite a high number of cases (more than 1,000) longitudinal analyses are also possible. A further opportunity is provided by merging data form national standard student assessment, which supports the analysis of relationships between innovation and school effectiveness.
On the basis of primary variables a number of composite indicators have been calculated. The most important among them are (1) the composite innovation activity indicator for both organisations and individuals, (2) the individual innovation behaviour indicator, (3) organisational dynamism indicators, and (4) the indicators of innovation sharing for both organisations and individuals. Some of these will appear later in the section on outcomes and analysis. One of the advantages of combining the subject and object approaches is that variables related with the subject (the innovating agent) and those related with the object (the specific innovation) can also be combined. For example, it is possible to test whether the value of indicators of sharing behaviour is higher or not in the case of subjects who present a specific innovation which has actually been shared.
A key element of the design of the questionnaires was that respondents (both organisations and individuals), after answering the questions about their general innovation activities/behaviour and contextual factors, were invited to identify one specific innovation they had created, and a number of questions were asked about this specific innovation. This design, as we have seen, is in accordance with what the Olso manual calls the combined subject/object approach. It can also be called “hybrid approach” as the subject- and the object-related focus is intertwined and cannot be easily separated when applying this approach.
The use of the hybrid or combined approach has raised a number of research design and methodology issues. When the subjects participating in the innovation survey are asked to name one of their innovations, and invited to answer questions about this single specific innovation subsequently, the unit of observation changes, although not radically. In the object-based part of the questionnaire the units of observation are not individual agents (subjects) but specific actions (objects) produced by them. The shift from the subject approach to the object approach is, however, less marked than in the case of surveys following the classical object approach where the same innovation (object) is observed as it is realized by several individual agents (subjects). Although the object-oriented part of our questionnaire has been focusing on the innovation and not on the innovator, the former remained strongly connected to the latter: we could continue to observe individual agents (organisations or people) through the innovations they have created. Similarly, although the attributes of specific innovations that appeared in our variables (e.g. their origin, age, change over time, and impact) related not to the subject (the innovator) but to the object (the innovation), we still had as many cases as observed individual agents. Datasets produced by such surveys continue to have the individual agents as cases, but one part of the dataset describes the attributes of objects and not those of subjects.
The combination of employer and employee data in the same dataset has increased complexity further. Working with databases combining the subject and the object perspective, on the one hand, and employer or workplace and employee perspectives, on the other, required special cognitive readiness to keep the appropriate focus of the analysis. This was supported by the model presented in Fig. 1 showing the specific analytical focus in function of the observed subjects and objects at both individual (employee) and organisational (employer) level.
Analytical focuses in combined subject/object and employer/employee innovation surveys
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
The data on specific or concrete innovations has enabled us to create composite indicators providing concise information on the key characteristics of these single objects. We constructed, for example, an “innovation value” indicator to describe the “weight”, “significance” or “seriousness” of particular innovations. This indicator allows to place every single innovation object on a scale ranging from “insignificant” to “serious”. This has been considered particularly important in the context of a survey focusing on “small innovations” initiated by organisational units or their employees.
The basic value of this composite innovation value index has been calculated as the simple mean of two primary variables: (1) the size of difference between the practice before and after the innovation was created, and (2) the size of the impact of the innovation on the effectiveness of work. The theoretical consideration behind this suggests that the “value” or “seriousness” of an innovation is higher if it fulfils two conditions: (1) it creates a practice that is significantly different from previous practice (novelty) and (2) it improves effectiveness (impact). This basic value was then weighted by five factors: (a) the survival of the innovation (those which were not abandoned got higher value), (b) the problem-driven nature of the innovation (c) the level of difference (those with a particularly high level of difference got higher value) and (d) the impact on effectiveness (those with a particularly high impact on effectiveness got higher values), and (e) initiatives to share the innovation with others. The impact on effectiveness was weighted higher than the difference between practice prior to and after the innovation was created. The weight of initiatives to share the innovation with others could increase the value of the indicator significantly (in case of having adopters within the country this could be 10%, while having adopters abroad could increase the value by 20%). Weighting has been based on theoretical considerations deduced from the Innova theoretical framework. The distribution of individuals (employees) and organisational units (employers) in function of the composite innovation value of the specific single innovation they selected and presented in the survey is shown in Fig. 2.
The distribution of individuals (left) and organisation units (right) in function of the composite innovation value index of the specific innovation they presented in the Innova survey
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
It is important to stress that the difference form previous practice, as well as the impact on effectiveness, was measured by self-reporting. Unit leaders (principals, heads of departments) and employees (teachers) were invited to answer the following question: “In your understanding, how much does the practice emerged from the chosen innovation differ from the previous practice?” The respondents were given four response options (1) “It does not differ at all” (2) “minimally” (3) “significantly” (4) and “to a very large extent”. The question, related to impact was as follows for organisations: “Please indicate to what extend has the innovation affected the success of your institution/organisation?” Individuals received an almost identical question: “In your opinion, to what extent has your practice been developed thanks to the selected innovation and how different is it from your previous practice?” In both cases five response options were offered which, in the case of organisational units was as follows: (1) “it has had a negative/damaging impact on the effectiveness of our work”, (2) “it did not affect the effectiveness of our work”, (3) “it has slightly improved the effectiveness of our work” (4) “it has significantly improved the effectiveness of our work”, (5) “it has greatly improved the effectiveness of our work.” Since unit leaders and employees answered almost identical questions, the composite innovation value indicator could be calculated for both organisations and individuals. In the section about outcomes and analysis several examples of analyses will be presented, most of them based on the use of the composite innovation value indicator.
The national context of exploring educational innovation processes
Before presenting and analysing the relevant outcomes of the Innova education sector innovation survey, it is important to explore the key features of the specific national systemic context in which the Innova project examined education innovation processes. Contextual factors – such as the bureaucratic or flexible operation of national systems and institutions, the level of social and professional support for innovation activities or the dominant patterns of thinking about innovation – influence not only innovation processes but also the way these processes may be explored and monitored. Politicians, development professionals, educational leaders and practitioners may encourage institutions and individuals to develop, adopt and share valuable practices, or encourage risk-taking and experimentation regardless of the value or seriousness of specific innovations emerging from this (subject perspective), and they may also support the development of specific novel practices and their spread and adoption regardless of who actually is involved in this (object perspective). Institutions and practitioners may be encouraged to invent new solutions to improve their daily practice which may result in large number of small grassroots level innovations, bit this might also be restricted by regulations and administrative control. While in some systems platforms to share best practices are developed and innovation sharing is supported by various mezzo-level institutional facilitators, in others no such mechanisms exist. All these factors may have significant impact not only on the general patterns of thinking about educational innovation and on how these patterns shape practice, but also on how innovation practices can be monitored using surveys.
For two decades (between the early nineties and the early 2010s) the education system of Hungary was one of the most decentralized systems in the OECD community. School level experimentation and innovations were strongly encouraged by regulations, incentives and massive capacity building. The OECD project called “Measuring Innovation” has evaluated classroom-level innovation performance of countries in several areas. According to its “overall composite education innovation index” – based on data from 2000 to 2011 – Hungary took the 6th place on country ranks – after Germany, Indonesia, Korea, Holland and Russia – significantly ahead of the OECD average (OECD, 2014).
This surprisingly high innovation performance can partly be explained by the high number of schools and higher education institutions participating in EU funded development interventions aimed at modernizing classroom level processes. These programs reached more than two thirds of Hungarian schools (Fazekas, 2018). A number of these interventions supported the implementation of curriculum-development programs encouraging teacher-led experimentation with new methods of organising learning. In some of them the condition of participation for schools and teachers was that they invent original new solutions and share them with others. Knowledge sharing school-networks and regional network coordination agencies were established, online platforms for publishing best practices in a standardized format were created and a market of good practices emerged, allowing schools to sell and buy innovative solutions. A number of schools participating in these programs later became “reference institutions”, prepared to provide capacity building services for other schools.
Although these innovation-boosting processes – gradually emerging after the accession of the country to the European Union in 2004 – were significantly slowed down following the 2010 parliament elections and a subsequent re-centralisation reform (Semjén, Le, & Hermann, 2018), a number of important drivers of institutional level innovation persisted. While the ownership of schools was transferred from local self-governments to the state, and central government control was strengthened also in higher education, the implementation of EU funded development programs was not stopped and support for institutional level innovation was partly maintained. For example, although the “innovation market” was abolished the online knowledge sharing platforms stayed alive and even new such fora were created. Similarly, while direct financial and managerial support for innovation sharing school networks was discontinued (e.g. the so called regional network coordination agencies stopped operating) the reference institution status was maintained and these institutions continued to receive formal governmental recognition.
Among the driving factors behind the relatively high level innovation activity in the Hungarian education system one has to mention the continuous and structured reflection on educational innovation. This is probably best illustrated by the fact that in 2010 an education sector innovation strategy, inspired by previous related OECD activities, was developed (Institute for…, 2011; OECD, 2016). This strategy positioned education sector innovation into the broader framework of public sector innovation and introduced the notion of National Education Sector Innovation System (NESIS). A few years later the NESIS strategy was updated (Balázs et al., 2015) making it possible for it to influence directly the development of a new teacher career promotion system. A specific teacher category called “master teachers” was created, and one of the requirements for access to this category became the demonstration of innovative professional practice through a professional portfolio. The NESIS strategy was recently updated again: a detailed chapter on digital innovation has been added (Halász et al., 2019). In these strategy documents the measurement of innovation was proposed as a key instrument of an education innovation policy. The specific national context, and particularly the continuous and structured strategic reflection on educational innovation has had a major influence on the design and the implementation of the Innova education sector innovation survey, especially on its focus on local and school level “small innovations”.
Exploiting the combined survey design: outcomes and analysis
In the section on survey design and methodological approaches we have presented the way the composite index of innovation value (“seriousness”) was created on the basis of data from the object-oriented part of our questionnaires. In this section we use this indicator together with variables (and other composite indicators) taken from the subject-oriented part of the same questionnaires. In most cases individual and organisational (workplace) level data will be combined or compared. All the analyses in this section are based on the combined databases of our 2nd and 3rd (pilot) data collections.
As we have seen, the actual value of the composite innovation value (“seriousness”) indicator of the Innova research depends on whether the selected specific innovation leads (1) to a significant alteration of practice and (2) to a significant improvement in effectiveness (as reported by heads of units and their employees). Our data allows the classification of both organisations and individuals on the basis of these two dimensions. We were using metaphors to describe the four different groups. As Fig. 3 shows the level of innovation activity of both organisations and individuals (calculated from data of the subject-oriented parts of the Innova questionnaire) is different in the four groups, identified on the basis of the characteristics of the specific innovations they presented.
The level of general innovation activity of organisations and individuals (subject approach) in function of the characteristics of the specific innovation they presented (object approach) (1–5 scale)
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
Before comparing different individuals/organisations in function of the value of the specific innovation they presented it might be worth taking a closer look at the reliability of our composite innovation value (“seriousness”) indicator. In our smaller pilot survey, based on face to face oral interviews, we asked unit leaders (N = 100) and their employees (N = 200) to present their selected specific innovation in a greater detail. On the basis of their stories our interviewers (trained to this task) were invited to evaluate the specific innovation stories using scales based on our theoretical models. As Fig. 4 shows sustainability and transferability were given the highest value by our interviewers, that is, these are the attributes that characterize the most strongly the presented specific innovations. More importantly, our data also shows that the highest differences between innovations having higher or lower values in our composite innovation value index appear in (1) their originality, (2) their cognitive complexity, (3) their “seriousness/significance” (as seen by our interview-making evaluators) and (4) the complexity of the problem targeted by the innovation. The lowest difference between the three groups appears in the sustainability, the extensiveness and the transferability of the selected specific innovation. This small scale testing gives sufficient confirmation of using our composite innovation value indicators for further analysis on our large-N population.
Different attributes of specific innovations presented by individuals (as evaluated by interviewers) in function of the value of these innovations as measured by the Innova innovation value composite indicator (1–5 scale)
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
Combining the subject and object approaches in the same questionnaire enables us to compare the values of the specific innovations produced by people/organisations presenting different levels or forms of general innovation activity and behaviour (as illustrated also by Fig. 3). Figure 5 shows the value of specific innovations identified by employees and by unit leaders (in the second case these are innovations produced by the unit/organisation) in function of their innovation activity. As the figure indicates those individuals and organisations that show higher level general innovation activity (measured using the subject approach) tend to present higher value (more “serious”) specific innovations, that is, innovations which (1) result in greater change of practice and (2) have a stronger positive impact on the effectiveness of their work.
The value of specific innovations indicated by employees and by unit leaders in function of their innovation activity
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
Innovation activity is a typical subject-based indicator: it shows how frequently individuals or organisations invent and introduce new solutions that change their practice and improve the effectiveness of their work. In this paper we do not present the way composite innovation activity indicators were calculated (for this see Halasz, 2018a): the focus is here on the composite indicator used to describe the value of specific innovations. We demonstrate that the characteristics of the object (the value of the specific innovation) is not independent of the characteristics of the subject (the innovator) who created or adapted the specific innovation.
Object-based indicators can be used, among others, to check the validity of subject-based indicators. If people and organisations who self-report higher level innovation activity, when are asked to present one of their own specific innovations – as demonstrated by Fig. 5 – tend to present something that is more “serious”, this shows that what they think about their innovation activity and behaviour in general (subject approach) is confirmed by the concrete example they actually present.
The combination of employer and employee data allows to look at individual innovators in the specific context of their work. In the Innova project one of the ways of linking employer level (organisational) and employee level (individual) data was the construction of four different constellations in function of the level of innovation activity of organisational units and their employees. We are using metaphors again to describe the four different constellations as shown in Fig. 6. This is again a case of combining both the subject (e.g. “innovation activity”) and object (e.g. “innovation value”) approach and the employer (organisational innovation activity) and employee (individual innovation activity) perspectives. The figure shows that individuals working in higher innovation activity organisations tend to present more “serious” innovations than those working in lower innovation activity organisations, and this is valid for both individuals demonstrating lower and higher general innovation activity.
The value of specific innovations presented by individuals in four different of constellations in function of the level of organisational and individual innovation activity
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
The inclusion of the diffusion dimension in the Innova research allows the comparative analysis of the innovation activity/behaviour of individuals who work in organisations that are less and more open to sharing innovations. In addition, the combination of the subject and object approach in the same survey instrument gives us insight whether this is reflected also in the nature of the specific innovations presented by the innovating agents. The subject-based part of our organisational questionnaire makes it possible to identify four different kinds of organisations in function of their sharing behaviour. As Fig. 7 shows, the specific innovations presented by people working in “giving organisations” have the highest value, although the difference between these people and those working in mutually sharing organisations is statistically not significant.
The value of specific innovations presented by individuals working in four different types of organisations in function of the sharing behaviour of the organisation
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
The subject-oriented part of our survey questionnaire lets us compare organisations along some other dimensions, as well. It is possible, for example, to compare organisational units showing higher and lower level dynamic capacities, that is, those that can be described less or more as “learning organisations”. Data show that the value of the specific innovations presented by individuals who work in organisations with higher level dynamic capacities is significantly higher than the value of specific innovations presented by individuals working in organisations with lower dynamic capacity (5.23 in the former while only 4.93 in the latter).
The inclusion of the diffusion dimension into a hybrid subject/object questionnaire allows a deeper analysis of innovation sharing on the basis of questions that can be answered precisely only when respondents have one single specific innovation in their mind. This is illustrated by Fig. 8 which shows the distribution of innovations in function of what their initiator said about sharing them and compares those mentioned by individuals with lower or higher general innovation activity. The latter attribute – as mentioned earlier – was calculated on the basis of answers given to the general questions of the subject-oriented part of our questionnaire while the three categories of innovations were created on the basis of answers in relation with the specific innovation from the primary variables of the object-oriented part of our questionnaire.
The distribution of specific innovations belonging to three different categories (in function of what their initiator said about sharing them) in groups of individuals with lower and higher innovation activity
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
Figure 8 illustrates well the shift from the subject to the object perspective. Here the unit of analysis is not the innovating agent (the organisation or the individual) but the innovation object created by them. The hybrid questionnaire generated data on a population of single innovations which – although each of them belonging to one well-defined specific subject – can be subjected to statistical analyses. The second and the third data collection round of the Innova project produced data about 1,560 innovations created by individuals and 1,240 innovations created by organisational units. The composite innovation value (“seriousness”) indicators, as mentioned earlier, were calculated on the basis of primary data about these specific innovations.
In the subject-oriented part of our questionnaire we also included a number of questions about the effectiveness of organisations. Both unit leaders and their employees were asked (1) to compare the effectiveness of their organisation to other similar organisations and (2) to compare the current effectiveness of their organisation to its past effectiveness. On the basis of this different composite organisational effectiveness indicators can be created reflecting both employer and employee views and inter-organisational and temporal (change in time) differences. As Fig. 9 shows the value of the presented specific innovations is higher in organisations that are seen effective by their leaders.
The value of specific innovations presented by organisations (head of units) and by individuals in function of effectiveness of the organisation as evaluated by heads of units
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
Merging the innovation survey databases with other databases allows further analyses of relationships between the value of the presented specific innovations and organisational effectiveness. In the case of the Innova project, for example, data from national student assessment were merged with the data from the innovation survey.
In the case of individuals the subject-oriented part of the Innova questionnaire contained a set of questions adapted from an instrument originally developed by de Jong and Den Hartog (2008), and later adapted to educational institutions by Messmann and Mulder (2012). The simplified version of this instrument, originally called Innovation Work Behaviour (IWB) scale, consists of 12 statements on various activities related with innovation. Respondents used a 1-7 Likert scale to indicate the frequency of these activities in their daily professional work. An exploratory factor analyses revealed two independent IWB factors: one is related with creativity and idea generation, the other with the implementation or realisation of ideas. On the basis of these two factors respondents could be grouped into four different cluster, named again with metaphoric labels. Figure 10 shows the average value of the specific innovations presented by people belonging to the four clusters.
The value of specific innovations presented by individuals in function of the individual innovative workplace behaviour (IWB) of respondents
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
Similarly to what we presented above, the relationship between the IWB indicator of individuals and the value (“seriousness”) of their observed specific innovation can also be analysed taking the organisational context into account. The data reveal, for example, that while “routinemen” and “innovators” working in more dynamic organisations present slightly more “serious” specific innovations than their counterparts working in less dynamic organisations, in the case of “managers” and “dreamers” the opposite could be observed. This may lead to interesting conclusions on the relationship between individual innovation behaviour and organisational context. It may be conclude that those who are able to combine creativity with implementation skills produce probably more serious innovations when they are placed into a more dynamic workplace, and those who do not possess either of these two sets of skills produce even less serious innovations if they work in less dynamic organisations. And, on the other side, those who possess only one of these two sets of skills are less likely to produce significant innovations even if they work in a more dynamic organisation.
The focus on single specific innovations allows us to explore the relationship between the “content profile” of innovations and their value. By content profile we mean the areas covered by the specific innovation. The Innova questionnaire specified 10 content areas and the respondents indicated whether the innovation they presented was covering the given area or not. Unsurprisingly the average value of the specific innovations is the lowest in those areas which are covered by the highest proportion of innovations and the highest in those areas where lower number of innovations are reported (see Fig. 11).
The proportion of organisational level innovations affecting specific areas and the value of innovations targeting these areas
Citation: Journal of Adult Learning, Knowledge and Innovation 5, 1; 10.1556/2059.2021.00042
In reality most innovations cover several areas simultaneously, so the interesting question is whether there are combinations that are accompanied by particularly high or low innovation values. This paper does not allow to elaborate this problem in details: only examples can be given here. In the case of innovations covering the design and implementation of lessons (which is very frequent), for example, the value of the innovation is significantly higher when it also covers the organisation (4.78) than when it does not (4.34). In other words, when – using the terms of the Oslo manual – process and organisational innovation components are combined in one innovation the probability of this innovation having a higher value is higher than in cases where only one of these areas are covered by the innovation. Similarly, when innovation combines element of lesson design and implementation with student evaluation its value is significantly higher (4.72) than in the case of innovations covering only student evaluation but leaving lesson design and implementation intact (4.29). An interesting feature shown by Fig. 11 is the breaking of the clear trend in the case of innovations covering external relations: these innovations seem to be more “serious” in terms of producing change of practice and having positive impact on effectiveness.
Conclusions
The hybrid or combined approach used in the innovation surveys presented in this paper has enabled us to explore the impact of a number of organisational and individual characteristics on the concrete (typically small) innovations people and organisations create in their daily practice in the education sector. This approach allowed (1) the measurement of the value (“seriousness”) of “small innovations”, (2) to distinguish less or more significant or “serious” innovations, and (3) to explore why certain people and organisations produce “more serious” innovations.
The combination of the subject/object approaches in a single questionnaire and the use of such a questionnaire in a matched employer/employee survey produces databases that offer unique opportunities to explore the dynamics of innovation activities and behaviours in the public sector where most of the innovations are small, and most of them are initiated and implemented at grassroots level by particular service delivery institutions and frontline practitioners. This form of data collection allows the observation of a population of large number of small innovations, and it facilitates the exploration of the “behaviour” of the “members” of this population. It supports, for example, the analysis of the life cycle, the modification, the “seriousness”, the impact or the diffusion of them. Our third (smaller-N, pilot) data collection demonstrated that this may allow to test complex models of innovation dynamics or to examine attributes such as sustainability, transferability, cognitive complexity or distance from mainstream solutions.
An important outcome of the Innova project is the demonstration of the relationship between the general innovation activity/behaviour of innovating agents and the value of the concrete innovations they produce in their daily work. Those showing higher level innovation activity tend to produce innovations with a higher value that probably lead to greater actual changes in their daily practice and to more positive impact on the effectiveness of their work. Innovations created by organisations showing higher level dynamic capacities tent to be more valuable than innovations created by less dynamic and less dynamic organisations.
References
Arundel, A. , Bloch, C. , & Ferguson, B. (2019). Advancing innovation in the public sector: Aligning innovation measurement with policy goals. Research Policy, 48(3), 789–798.
Arundel, A. , & Huber, D. (2013). From too little to too much innovation? Issues in monitoring innovation in the public sector. Structural Change and Economic Dynamics, 27, 146-149.
Balázs É. , Fazekas Á. , Fischer M. , Győri J. , Halász G. , Kovács I. V. , et al. (2015). „Okos köznevelés”. Javaslat a Nemzeti Oktatási Innovációs Rendszer stratégiájának kiegészítésére [Smart public education. A proposal for the complementing the Strategy of the National Education Sector Innovation System]. Budapest: ELTE PPK.
Bugge, M. M. , & Bloch, C. W. (2016). Between bricolage and breakthroughs—framing the many faces of public sector innovation. Public Money & Management, 36(4), 281–288.
De Jong, J. P. , & Den Hartog, D. N. (2008). Innovative work behaviour: Measurement and validation, Working paper, Zoetermeer: EIM Business and Policy Research.
Dostie, B . (2018). The impact of training on innovation. ILR Review, 71(1), 64–87.
Fazekas Á . (2018). The impact of EU-funded development interventions on teaching practices in Hungarian schools. European Journal of Education, 53(3), 377–392.
Fuglsang, L. (2010). Bricolage and invisible innovation in public service innovation. Journal of Innovation Economics, 1, 67–87.
Gallouj, F. , & Savona, M. (2010). Towards a theory of innovation in services: A state of the art. In F. Gallouj , & F. Djellal (Eds.), The handbook of innovation and services. A multidisciplinary perspective (pp. 27–48). Cheltenham: Edward Elgar.
Gault, F. (2018). Defining and measuring innovation in all sectors of the economy. Research Policy, 47(3), 617–622.
Georgsen, M. , Beard, A. , Ilomäki, L. , Mor, Y. , & Koskinen, T. (2015). The teacher's role in educational innovation. Elearning Papers, (44).
Godin, B. (2004). Measurement and Statistics on Science and technology: 1920 to the present. London and New York: Routledge.
Halász, G . (2018a). Measuring innovation in education: The outcomes of a national education sector innovation survey, European Journal of Education, 53(4), 557–573.
Halász, G . (2018b). Teacher learning and innovation: The case of Hungary. A paper prepared in the framework of the „Belt and Road Education Research Project” initiated in 2017 by the Chinese National Institute of Educational Sciences. Forthcoming.
Halász, G. (2020). Measuring innovation in education with a special focus on the impact of organisational characteristics. Forthcoming.
Halász, G. , Fazekas, Á. , Fischer M. , Horváth L. , Kovács I. V. , & Lajos, P. (2019). DigiNOIR. Stratégia-javaslat az oktatási ágazat innovációs stratégiája (NOIR stratégia) oktatástechnológiai pillérének kibontása [DigiNOIR. Strategy-proposal elaborating the technoilogy pillar of the education sector innovation strategy]. Budapest: DPMK.
Hargreaves, A. , Lieberman, A. , Fullan, M. , & Hopkins, D. W. (Eds.) (2014). International handbook of educational change. Part two. Springer.
Høyrup, S. (2012). Employee-driven innovation: A new phenomenon, concept and mode of innovation. In S. Høyrup , M. Bonnafous-Boucher , C. Hasse , K. Møller , M. Lotz (Eds.), Employee-driven innovation. A new approach to innovation (pp. 3–33). London: Palgrave Macmillan.
Institute for Educational Research and Innovation (2011). Strategy proposal for the development of the Hungarian national education sector innovation system .http://ofi.hu/sites/default/files/ofipast/2011/05/8.1.-Vezetoi_osszefoglalo-EN.pdf).
Kärkkäinen, K. (2012). Bringing about curriculum Innovations: Implicit approaches in the OECD area. OECD Education Working Papers, No. 82, OECD Publishing.
Lippke, L. , & Wegener, C. (2014). Everyday innovation – pushing boundaries while maintaining stability. Journal of Workplace Learning, 26(6/7), 376–391.
Messmann, G. , & Mulder, R. H. (2012). Development of a measurement instrument for innovative work behaviour as a dynamic and context-bound construct. Human Resource Development International, 15(1), 43–59.
NESTA . (2007). Hidden innovation. How innovation happens in six ‘low innovation’ sectors. Research report. London.
OECD . (2014). Measuring Innovation in education: A new perspective, educational research and innovation. Paris: OECD Publishing.
OECD . (2015). The innovation imperative: Contributing to productivity, growth and well-being. Paris: OECD Publishing.
OECD . (2016). Innovating education and educating for Innovation. The power of digital technologies and skills. Paris: OECD Publishing.
OECD/Eurostat (2018), Oslo manual 2018: Guidelines for collecting, reporting and using data on innovation (4th ed.). The measurement of scientific, technological and innovation activities. OECD Publishing, Paris/Eurostat, Luxembourg.
Semjén, A. , Le, M. , & Hermann, Z. (2018). The goals and consequences of the centralization of public education in Hungary. Acta Educationis Generalis, 8(3), 9–34.
Van de Ven, A. H. , Angle, H. L. , & Poole, M. S. (2000). Research on the management of innovation. The Minnesota studies. Oxford University Press.
Vincent-Lancrin, S. , Jacotin, G. , Urgel, J. , Kar, S. , & González-Sancho, C. (2017). Measuring innovation in education: A journey to the future. Paris: OECD Publishing.
Vincent-Lancrin, S. , Joaquin, U. , Soumyajit, K. , & Gwénaël, J. (2019). Measuring Innovation in education 2019: What has changed in the classroom? Educational research and innovation. Paris: OECD Publishing.
