University monitoring research : experience and results

Stepan Gzhytskyi National University of Veterinary Medicine and Biotechnologies Lviv, Pekarska Str., 50, Lviv, 79010, Ukraine. Tel.: +38-098-708-08-95 E-mail: smolinskao@gmail.com Smolinska, O., Drach, M., & Dzyubynska, K. (2018). University monitoring research: experience and results. Scientific Messenger of Lviv National University of Veterinary Medicine and Biotechnologies. 20(86), 153–160. doi: 10.15421/nvlvet8630


Introduction
The conducting of monitoring research at universities is very diverse both in the terms of its objectives, and from the standpoint of appropriate technologies usage.Obviously, the goals, tracking the dynamic changes of the quality of higher education and educational activity, have recently become vitally important, therefore, the monitoring research is becoming an internal university tool to define the quality of university education.Analyzing the current situation at Ukrainian universities, we concluded that it resembles the situation, which was at the leading European and American universities in the early 1990s, when the quality of university education became the major issue in the sphere of politics, economics and management.According to Harvey Lee and Askling Berit (2003), these changes were crucial and emerged as a result of combination of different elements: the resigning of commission, the new state sector management, restrictions in funding, increase in the number of students, etc. -«all were manifested in claims for external quality monitoring and catalytic function of monitoring for internal improvements within institution» (Harvey and Askling, 2003).
Actually, real autonomy of universities, which changed the attitude to monitoring and became a new stage of university development, was, first of all, the introduction of the credit-modular system (CMS) (Order of Ministry of Education and Science of Ukraine (MESU) № 774 «On the introduction of credit-modular system in the educational process») since 30.12.2005, and then the adoption of credit-transfer system (CTS) (Order of MESU № 943 «On the introduction of European credit-transfer system at higher educational institutions of Ukraine», since 16.10.2009)Initially, the recommendations, that were adopted together with the order of MESU № 774 (2005), emphasized a need to take a number of measures, aiming at the stimulation of the process of formation of internal quality control elements, including the creation of the internal monitoring system to assess the quality of specialist training.The initial number of universities, during the first stage of the experiment (2003)(2004), was 59, including 7 with regional status, then during the second stage (2005)(2006)(2007)(2008) -other universities started the reorganization of educational activities, according to the order of MESU № 774 (2005) Monitoring research, which is an integral part of an internal quality control system of education in higher educational institution, has partially different meaning in most Ukrainian universities, comparing with the similar foreign studies.This difference is both the result of different orientation of internal quality systems, and their different content.It would not be correct to state that there have not been any quality control systems yet; it is a necessary to mention the Soviet system of CSPQM (complex system of product quality management), the rudiments of which we are still observing, such as: traditional Soviet-style open lessons, mutual and control attending of lectures, that were adopted by Ukrainian system of higher education.The reasons for such «conservation» are numerousone of the main is considered to be the orientation of qualitative educational parameters not on the students or employers, but on universities as organizations and their management.This happened due to the delay in the cancellation of a number of regulations concerning the organization of the educational process and came into action at the beginning of 90-es and until recently.Simultaneously, a lot of Ukrainian universities, especially those that position themselves as flagships of change, are creating such internal quality control systems that are aimed at students and their future career, besides, these systems are closer to adjustment than regulation and do not have the signs of repression.Currently, active changes are taking place, but the already gained results of our research are fixing the current state of the problems.The aim of our article is to sum up and analyse the results of eight-year monitoring research of the quality of organization of educational process at Stepan Gzhytskyi National University of Veterinary Medicine and Biotechnologies Lviv.
There are three types of informational resources of monitoring research of higher education in Ukraine: the information provided by universities on their websites, that are, mostly, regulations on monitoring and / or monitoring groups in higher educational institutions (available at appropriate sites), scientific publications (Hrynevych, 2010;Olendr, 2011;Hotsul et al., 2012;Rakov, 2013;Plysenko, 2016), textbooks for students of certain specialities, scientists, practitioners (Tsekhmistrova, 2004;Pasichnyk and Riezina, 2011).
Other sources state the fact that the monitoring research in higher education sets some other tasks.In particular, O. Belash et al. (Belash et al., 2015), firstly, determine another reference point of the education quality system -stakeholders and, secondly, the object of their potential satisfaction is the quality of the educational program and the environment of its implementation.One more possible option, regarding the internal quality system and its monitoring, is to conform it according to the ISO standards, this issue is studied by D. Dugarova et al. (Dugarova et al., 2016).
The analysis of exclusively pedagogical aspects of internal quality systems is carried out in OECD-Institutional Management in Higher Education (IMHE) project ( 2008), where the institutional subjects that form the quality of education (teachers, institutional leaders, students) are defined, as well as the main units of the quality are distinguished and the recommendations are given, namely: «Ensure that the institutional policy on quality teaching is understood and implemented properly by faculty members.Disseminate a quality culture in the whole institution and facilitate collaborative work and information fluidity.Reconsider their reflection role in addition to the more technical one.Combine research in educational sciences with the definition of practices.Experiment to develop new measurement and evaluation methods.Be receptive and enhance communication tools to gather teacher and student suggestions.Keep an open-oriented approach towards external inputs and good practices examples, creating a communication network with quality assurance agencies and external partners» (p.76).
The study of the challenges, which British universities face, is due to the expansion of private educational institutions and the recommendations on the improvement of the quality of education, which was carried out by the University Alliance ( 2014), where the monitoring is a technical tool for the assessment of general educational situation in the system.
The article by I. Huet et al. (Huet et al., 2010), is thematically close to our article, because it also considers various possibilities for the creation of a model of the university high-quality educational system, its components, subjects and regulatory mechanisms, based on the monitoring research -the Quality Assurance System.

Methods
The monitoring research of the quality at Stepan Gzhytskyi National University of Veterinary Medicine and Biotechnologies Lviv has started in 2008 and it is annually held during the spring semester.The means of monitoring research are anonymous questionnaire with questions of open and close types.Every year the amount of participants of the survey are students from different courses, approximately 900 ± 100, except for the firstyear students.This number of respondents, on average, amounts to one third of the total amount of full-time students.The study of the opinion of the first-year students on the organization of the educational process is inefficient, due to their low level of awareness, and secondly, because of the implementation of transitional organizational forms in their studies (for example proseminar) and the specific structure of their curriculum, where social sciences and humanities are present, which help to soften the transition from secondary to university education; the didactic principles of consistency and continuity are actualized.

Results
The research, aimed at monitoring of the quality of higher education and related activities, which was held at Stepan Gzhytskyi National University of Veterinary Medicine and Biotechnologies Lviv, evolved and passed several conventional stages: the first one -pilot (2008-2009 years), the second -the research on the organization of the studying process, according to the credit-modular system (2010-2011 years), the third -the research on the organization of the studying process, according to the credit-transfer system (2012-2015 years), the fourth -the research on the organization and the quality of the educational process (2016).
Each stage met certain goals and had definite results.The first stage analysed the learning process from the students' point of view.It interpreted the obtained results and transferred them into the sphere of communications «university-student».
The second stage monitored real efficiency of educational process organization according to the creditmodular system, aiming at regulatory correction (30.8% questions).Moreover, in the survey, special attention was paid to the issue of providing the quality of educational activities (53.9% questions) and the prevention of corruption (15.3% questions).Open question was formulated in the following way: «What would you like to change at your university and the faculty?» The structure of questions was changed on the third stage: questions on the organization of educational process -33.3%, quality -50%, prevention of corruption -16.7%.Three questions -on professional choices and employment -were added.The following open questions are: «Could you specify the disciplines, which you consider inappropriate for your professional training?What subjects would you like to introduce into the curriculum, or to increase the number of hours to improve your professional training?Your suggestions on the improvement of the learning process at the university».The questionnaire was slightly modified in 2014, concerning the open questions: two of them were united in one (about subjects) and those, related to the professional choice, were removed, as they became a part of the marketing department study.
On the fourth stage the structure and content of the questionnaire were modified, some questions were added.They concerned the study of the level of subjective perception of the type of educational space, prevailing at university, especially the degree of student involvement in achieving the educational results.Thus, the questions, regarding the organization of educational process, made up 36.8%, the quality -31.6%, the prevention of corruption -5.3%, the subjectivity of students -26.3%.First of all, these changes were intend to implement the university's desire to build an efficient communication with students and, secondly, to assess the development of the declared student-oriented approach, thirdly, to determine the prospects of the strategy formation and tactics to build internal quality system.
Finally, we obtained the answers to the cross-cutting issues of the 2-4 stages (pilot stage questionnaire was structured in a different way): -concerning the organization of educational process («Does CTS motivate to work systematically?»,«Which difficulties in your training are caused by CTS?», «What form of final term control is the most appropriate for you?»); -concerning the quality of educational activities («Assess the level of teaching at the faculty in grades», «Does the level of educational literature provision correspond to the demands that are put forward to your educational results?»); -concerning the prevention of corruption («Do you feel that your teachers are interested in your educational success?»,«Have you ever had conflict communication with teachers?»,«Do teachers follow ethical standards in teaching and communication?»).
The results of the survey are given below.We investigated the results and revealed the following tendencies: the issues of organizational character had general positive dynamics of strengthening the stimulating effect of credit-transfer system till 2014 and the subsequent weakening (Figure 1), which may be the result of misinterpretation of the abolishment of the order of MESU № 774 (2005) «On the adoption of credit-modular system into educational process» in 2014.The lack of media and legal literacy both among the students and among the teachers has led to the distortion of information and, therefore, the authority and effectiveness of CTS were wrongly devaluated.
Another issue, aimed at the monitoring of educational process organization at the university, was to find out the essence of the difficulties, which students should overcome, to achieve good educational results (Figure 2).The following tendencies are clearly seen: the reduction of abstract work significance (from 24% to 7%), the increase of opportunities to speak up your mind during the lesson (from 10%, to whom it was problematic, to 3%), high frequency of control tests was sometimes assessed as bigger complication and sometimes not (tendencies were not observed).

Table 1
The preferable form of the term control of educational results Despite the fact that the first question (Figure 1) demonstrated some reduction in stimulatory function of CTS, the tension of work is the growing factor, which significantly affects the sense of difficulty in achieving individual educational goals, during the semester (from 42% to 47%).While determining the priority form of the final term control (Table 1), students gave their preference to tests (on average 45%) and to the different combined forms (29%), and moreover, since the introduction of compulsory term examination (2015), the importance of tests has decreased, but of other combined formsincreased.
The second set of questions is dedicated to the subjective assessment of the quality of the educational process by the students.

Figure 3. Students' assessment of the quality of educational activity at university
The practice of assessing the quality of teaching in grades is familiar to a student, that is why the results adequately reflect their real opinion (Figure 3).In prospect, the comparison of students' success and the marks, which they give to teachers, may be also of great scientific interest.To conclude, the absolute index fluctuates within 98%, ratings from grades 3 to 5 (scale from 2 to 5), whereas the fluctuation of 4 and 5 marks is from 65% to 82%.A significant decrease in quality was also connected with the introduction of compulsory examination in 2015 when both teachers and students were adapting to new organizational conditions.The study of the issue of the correlation between the level of educational literature provision and the demands that are put forward to the educational results of students is determined by the need to monitor the conformity between studying methods and means.The analysis of the students' responses (Figure 4) confirmed that such a conformity exists.However, the growth in 2014, followed by the decline, demonstrates the dependence of this index not only from the work of the department, library, the students themselves, but also from purely technical factors (for example, the connection to the internet of all the academic buildings and dormitories of the university was fully completed in 2012).At the same time it coincided with a change in educational standards and approaches to the publication and distribution of educational materials.This question was changed in the questionnaire form after 2016.

Figure 5. The evaluation of direct communication with teachers, done by students
The following set of questions directed at the prevention of corruption included three questions: «Do you feel that your teachers are interested in your educational suc-cess?»,«Have you ever had conflict communication with teachers?»,«Do teachers always follow the ethical standards in teaching and communication?» (Figure 5).These questions, formulated in such a way, made it possible to detect the level of risks of conflict and corruptive behavior.In general, students felt some decline of teachers' interest in their educational success, the reason, except for the objective ones, was the increase of their selfsufficiency.Concerning the conflict communication, the average index amounted to 63% over the years.However, it was done without further examination and classification of conflict types.In the future this issue will be studied with the help of separate survey by social and psychological services, as it is not illustrated in this article.Nevertheless, the level of the teachers who keep to the ethical norms is rather high (74%) and demonstrates the stability ± 6%.
Another result was the determination of such index as: the necessary to combine education and work, it was not included into general summary, as was obtained only during last three years.It increased from 36.7% in 2014 to 46.6% in 2016.This result also correlates with the results of the study of migration tendencies, which was conducted in our university and other Lviv universities in 2010/2011 academic years (Stybel et al., 2015).Comparing these results with the similar ones, obtained at Kharkiv National University named after V.N.Karazin (Nikolayevska, 2017), we can state that the level of migration disposition is higher among Lviv students in general, and, particularly, among the students of our university, and has a different structure (at the time of the research, the possibility to study was more important for the citizens of Lviv (88.24% vs. 33%) and to work (87.8% vs. 52%), showing higher index of potential migrants (16.02% vs. 2%)).
Consequently, the monitoring at Stepan Gzhytskyi National University of Veterinary Medicine and Biotechnologies Lviv is only one of the possible means of internal system formation for providing the quality of education and educational activities, which are its balancing components.According with the annual results of the study, a lot of management, personnel, economic decisions were made, both at the group and departments levels, as well as at the university level.

Discussion
Thus, the conduction of the eight-year monitoring research and its results have facilitated the implementation of administrative innovations to solve tactical problems by means of the development of horizontal communication.The experience of the similar research (Annienkova, 2010;Nikolaievska, 2017) affirms that both the content of the questionnaires, and the results correlate with the specific aims of monitoring research of the university where it was conducted.The criteria of control also differ, for example Annienkova I.P. (2010) attributes the following criteria to the index of university-level: «the rating of the higher educational institution among other universities; the percentage of graduates who are employed in the field of their specialty; the percentage of graduates who work in the field of their specialty in 1, 3, 5 years after graduation; the assessment of graduates' readiness for professional activity as a supervisor/ top manager (at the high, medium, low level); self-assessment of the graduates' readiness to the professional activity (high, medium, low); salaries of graduates» (Annienkova, 2010).At the same time, the mark of the quality of teaching can be compared: for instance, at Kharkiv National University named after Karazin, students assessed the quality of teaching as 4,3 points during the period of 2002-2010 (Nikolaievska, 2017), while our research has documented an average grade -4.0, for the last 7 years (Figure 3).The thematic questioning «Teacher from the students' point of view» became the most widely spread, for example, at National Technical University of Ukraine «Igor Sikorsky Kyiv Politechnic Institute», Kyiv National University named after Taras Shevchenko, Kyiv University of law, Ivan Franko Lviv National University and others; their mission was to become the instruments in the implementation of personnel policy, the factor of individual teacher's rating, such kind of research was also carried out by us, but the results were not presented in this article.Besides, it's impossible to ignore the general approach to the preparation of questionnaires, their estimation as solely sociological tools (Hapon and Dubniak, 2009;Shvets and Turba, 2011;Pilov and Svizhevska, 2014).The publishing of the research results is usually done at the university sites, in such a way, they are partially isolated from the general scientific discussion, and therefore require more active generalization, at the same time ethical boundaries of such generalizations, concerning universities, are not completely obvious.Although, the information about the overall level of student's satisfaction with the organization of educational process and their requirements to the teachers' personality are subjectivelyoriented.

Conclusions
Thus, the purpose of this article was to sum up the results of the monitoring research (2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015)(2016), which was carried out at Stepan Gzhytskyi National University of Veterinary Medicine and Biotechnologies Lviv.The monitoring tasks were to investigate the actual content of the reorganization that was held in connection with the introduction of new systems of organization of educational process and the reflection of their results on the formation of internal system of quality of education.Every year the participants of the survey are students from different courses, approximately 900 ± 100, except for the firstyear students.According with the analysis' results, the stages of monitoring research were distinguished (4 in total), stages 2-4 were comparable with each other and on their basis the following thematic groups of questions were traced: questions on the organization of educational process; questions on the provision of the quality of educational activity and also questions on the corruption prevention.
The general tendencies in the students' answers to such questions were analyzed:  «Does CTS motivate to work systematically?»Mostly these were positive answers (85 ± 5%).
 «Which difficulties are caused by CTS in your training?»The gradual decrease in the significance of the problem of large number of abstract work, students' inability to express themselves during practical classes can be noticed; the issues of the frequency of control tests and tense work during the semester remain, more or less, stable; the number of answers «Other» increased and this is the reason why in the future the students will be offered to give their individual answers instead.
 «What form of final term control is the most appropriate for you?»It was found out that the students prefer tests (45 ± 11) and different mixed forms of tests (29 ± 9).
 «Assess the level of teaching at the faculty in grades».GAP was 4,02.
 «Does the level of educational literature provision correspond to the demands that are put forward to your educational results?»The answers to that question were related to a number of objective factors, for example the connection of all the university buildings and hostels to internet.The general tendency is the following: the level of provision fluctuates due to the changes in systems of educational process organization.
«Do you feel that your teachers are interested in your educational success?»«Have you ever had conflict communication with teachers?»«Do teachers follow ethical standards in teaching and communication?».These three questions were aimed at the detection of communicative drawbacks and the risk of corruption rise.The following results are obtained: the students feel that teachers are interested in their educational results (67%); 63% of the respondents faced communicative conflict situations; 25% complain about the violation of ethical standards.
The prospective of further research can be seen in two directions: elaboration and verification of tools for further internal monitoring research of the quality of education; the preparation of common ethical approaches to such types of the research and promulgation of these results.
. Stepan Gzhytskyi National University of Veterinary Medicine and Biotechnologies Lviv launched a new stage of individual quality regulation of educational indices, based on the «Provisional Regulations on credit-modular system of educational process» (2006) and started in 2006/2007 academic years.

Figure 1 .
Figure 1.The assessment of CTS as a stimulating educational factor (2010-2016 years)

Figure 2 .
Figure 2. The specification of the content of educational factors that cause difficulties (2010-2016)

Figure 4 .
Figure 4. Students' evaluation of the adequacy between the quality of educational literature provision with own educational results