PROJECT MANAGEMENT METHODOLOGY IN THE PRACTICE OF EVIDENCE-BASED DEVELOPMENT, EVIDENCE-BASED POLICY
Olga FIOFANOVA
Russian Academy of National Economy and Public Administration
under the President of the Russian Federation
Moscow, Russia, Pr. Vernadscogo, 84
ORCID 0000-0003-3004-8067
Financing: Russian Foundation for Basic Research, project grant No 19-29-14016
Abstract
It is analyzed: how the principles of evidence-based management, evidence-based policy based on big data analysis are integrated with project management methodology (ISO 21500: 2021). The results of changes in the management practices of state projects and programs are presented. The tasks of the competence transition from big-data to smart-data for data-management and evidence-based policy are formulated. The practice of implementing the methodology of evidence-based management, evidence-based policy is associated with the implementation of the concept of "Open Data", the National Data Management System in Russia, the development of the technological infrastructure of big data and digital data analytics services. Our research under the RFBR grant No. 19-29-14016 "Methodology for Big Data Analysis and Its Integration into Professional Development Programs for Executives" revealed how data analysis and evidence-based management principles are used in project management activities for state development programs. At the strategic management level: strategic performance indicators, infrastructure and data sources are defined. At the level of project management activities, sources of alternative data analysis are determined based on feedback from citizens (public policy). At the level of project life cycle management, it is determined which data sets will indicate the dynamics of value development and the effectiveness of products in the portfolio of projects of state development programs. At every level of evidence-based project management, this requires data competencies from leaders.
Keywords: project management methodology, evidence-based development, evidence-based policy, impact investing, big data analysis methodology
1. Introduction
The practice of implementing a management methodology based on data analysis is associated with the implementation of the Open Data Concept, the National Data Management System in Russia, the development of the technological infrastructure of big data and digital data analytics services.
The practice of implementing project management is associated with the need to ensure development through differentiation in state programs of project and process components (Decree of the Government of the Russian Federation "On the transfer of the state program" Development of education "to project management") [1].
The new Regulation on the management system of state programs of the Russian Federation [2], which establishes the rules for the development, implementation, monitoring and evaluation of the effectiveness of state programs of the Russian Federation (comes into force in 2022), updates new monitoring models congruent with the set of planned results, as well as actualizes demand on the methodology of data analysis (methodology for calculating indicators) and digital data analytics services based on artificial intelligence technologies. The regulation provides for “the allocation in the structure of state programs - PROJECTS, determined, formed and implemented in accordance with the Decree of the Government of the Russian Federation of October 31, 2018. No. 1288 "On the organization of project activities in the Government of the Russian Federation" (Regulation on project activities).
Thus, the State Program is a tool for strategic development management, a tool for budgeting development directions, a tool for consolidating communities participating in development projects.
The analysis of data as conditions and results of development is organized in the state automated information system "Management", which collects and processes data, analyzes the implementation and evaluates the effectiveness of the implementation of state programs and their structural elements (PROJECTS).
In the analytical information system "Portal of state programs" (www.programs.gov.ru) [3], publicly available information is posted on the results of data analytics and on the assessment of the effectiveness of state programs.
Probably, the new management practice actualizes the need for the development of data-competencies among educational leaders. The project competencies of managers have been evaluated and certified for a long time [4], and competitive practices [5] make it possible to identify the most effective organization of Project offices responsible for the implementation of state projects and state programs and analysis of their effectiveness. But data competencies are not included in certification and competency assessment systems. It is these competencies that change the competence structure of management activities.
In many countries of the world - top 100 according to the Networked Readiness Index, digital environment standards and data analysis standards are developed. Such standards make it possible to assess the technological maturity and maturity of the management system in the implementation of government projects and government programs.
In 2021, Rosstandart introduced the Big Data terminology standard [6] GOST “Information Technologies. Big data. Overview and vocabulary ", which is identical to the international standard" Information technology - Big data - Overview and vocabulary ". After terminology standards, technology standards are enacted. But already now in the terminological standard the sequence of actions in working with big data is fixed: “data analytics - receiving, collecting, checking and processing data, evaluating them, visualizing and interpreting”, “data variability - the format and structure of data data, semantics and quality of the data array , variability of values ”[6].
If technologically working with big data is standardized in international and national standards, and standardized technologies imply the universality of working with big data, then with regard to data analytics in the field of education - methodologically working with data is more complicated.
This complexity is primarily due to the fact that in education as a field of human development, not only the Perfomance Management Systems methodology is applied, but also the Human Capital Management Systems methodology, which characterizes the optics of humanitarian systems, digital humanities, big-data development analytics person.
- Problematization, research issues
The development of the practice of evidence-based management is associated with the need to justify project decisions on the development of the state, society, and economic sectors. It is also connected with the need for reasonable project budgets and an assessment of their investment efficiency, the impact impact of investment projects.
In the conditions of a developed system of generation of project management standards[1] [5], investment standards[2] [4], there is clearly an insufficiently developed methodology and technologies for big data analysis, as well as the integration of data analysis methodology with project management methodology and impact investment methodology.
In fact, such an integrated methodological framework is the basis for standardizing activities in the field of evidence management, evidence policy, both at the state federal level and at the territorial regional level, in the sectoral and organizational-institutional aspect.
In this regard, a number of research questions arise: 1)how are the project management standard and big data analysis technologies and the principles of evidence-based data management methodologically integrated? 2)how does the module of competencies of a project management specialist, a project manager develop in the conditions of the need for proof of project decisions based on data analytics? 3)how do the management decision styles of project managers change in practice in the conditions of working with big data?
1.2. Methodology and methods of research.
The methodology of the study was the concept of project management [3][6]; the concept of the national data management system [4] [5]; the methodology of big data analysis [5] [3]; the concept of management decision-making styles [6] [8].
Research methods:
a) the method of structural and functional analysis of project management activities of state programs / development programs;
b) the method of content analysis of public reports of the governing bodies;
c) method of research of styles of management decisions;
d) a method of theoretical and methodological analysis of project management standards and generated standards for big data analysis for evidence-based management.
2. Analysis of the development of the concept and practice of evidence-based management and evidence-based policy
In the conditions of a developed system for the formation of project management standards [7; 8], investment standards [9], the methodology and technologies of big data analysis for the management of state projects and state programs are not sufficiently institutionalized. In fact, the methodology is the basis for the standardization of activities in the field of evidence-based management, project management based on data.
In the field of education, data analytics standards are technologically linked to the standards of the digital educational environment, which determines the architecture of data on human development, the conditions of his education, and are ideologically linked to the principles of human development.
In this regard, a number of research questions arise:
1) how are the project management standard and big data analysis technologies methodologically integrated in management activities in the context of the implementation of the principles of evidence-based management based on data?
2) how does the manager's competence assessment model change in the context of working with big data and the tasks of implementing evidence-based decisions based on data analytics?
3) do the styles of management decisions among managers change in practice in the context of working with big data?
4) how are data architecture and assessment tools changing, monitoring education development, what is the digital data infrastructure?
The research methodology was made up of the methodology for the analysis of big data in education [10]; the concept of project management [11]; the concept of a national data management system [12]; big data analysis methodology [13]; the concept of styles of forming managerial decisions [14].
The concept of evidence-based governance and evidence-based politics is studied in science and also institutionalized in the practice of governance. In the countries of the world, special institutions are being created as elements of the ecosystem of evidence-based management - management based on data analysis.
These are technological infrastructures and data analysis services, and institutions for the professional training of competent personnel, institutions of expertise and evidence-based policies. The USA has the Chief Evaluation Office, as well as The U.S. Commission on Evidence-Based Policymaking (CEP). СEP organizes the implementation of national EBPM projects.
In Japan, the Council for Science, Technology and Innovation (CSTI) is responsible for collecting and analyzing data and coordinating the e-CSTI platform as a technology infrastructure for evidence-based government project management. The evidence-based policymaking principles were approved by the Cabinet of Ministers.
A technological data infrastructure is being developed in Russia and the Concept of the National Data Management System has been approved. Evidence-based policymaking principles are integrated into the activities of governing bodies.
In the UK, there is Administrative Data Research UK (ADR UK), whose analytical recommendations form the basis of projects. ADK UK accredits data analysts as evidence-based policy professionals.
The Corporation for Social Research is established in Canada, which conducts data analysis and evaluates the effectiveness of projects based on data analysis.
In the study [10], we analyzed the development of data management in the countries of the world. We compared the E-Government Development Index - E-Government Development Index, EGDI - and the Open-Data / Open Government Index.
2.1. Factors determining the development of evidence-based management practices in project management
A study of evidence-based management standards, national databases of analytics and statistics revealed the following:
1) trends in the implementation of management policies based on data and the ideology of open data are associated with ensuring the organizational coherence of institutions that provide educational data through legal regulations and laws of countries, as well as with the development and implementation of integral digital technologies that allow integrating different types of educational data through their digital programs. processing for building a system of connections and forecasting opportunities;
2) countries implementing a policy of human development through projects of state programs for the development of education organize the collection and analysis of data based on two Data-Analys methodologies: activities; b) institutional methodology, in which the nature of the data is related to reports on the conditions of educational activities, reports on the implementation of educational development programs. Subsequently, these two variables in educational data are compared in the search for correlations for evidence-based analytics as the basis for making effective decisions;
3) countries that implement the policy of openness of educational data and openness of data analytics services on digital educational data platforms, as a rule, implement an ethical policy by explaining to users the ethical protocols of working with educational data;
4) according to the management structure, the institutions responsible for ensuring the functioning and development of digital databases (repositories) of educational analytics and statistics data are directly subordinate to the government of the country or a sectoral ministry - the Ministry of Education;
5) countries with a developed technological infrastructure of analytics and statistics of educational data and highly functional repositories of educational data integrate for processing and interpretation educational data of the institutional, national and international levels of assessment of educational results related to school systems for assessing educational results, national certification exams and the results of international quality studies education. This allows for timely alignment and harmonization of education monitoring systems and assessment tools.
2.2. Results of the study of management decision styles in the implementation of development projects and programs, analysis of the principles of evidence in public reports
With regard to the sectoral aspect - education - in the study, we analyzed:
a) what data in education do heads of the federal, regional level in the field of education work with in managing state programs for the development of education and projects in the structure of state programs;
b) what data do the heads of educational organizations work with when implementing development programs for educational organizations and projects in the program structure;
c) what data do teachers work with?
The results of the study of pedagogical work with data, methods and technologies of data analysis in various information and analytical systems and educational platforms are presented by us in publications [11]. Both the existing infrastructure of education and human development data and the projected infrastructure and architecture of educational data reflecting new indicators of human development in post-industrial society are analyzed.
With regard to data analytics at the managerial level, the infrastructure and services for data analysis are developing in the field of education, new objects of managerial analysis are emerging, and the monitoring system is developing.
The data architecture of the state program for the development of education is constructed in the integration of the areas of analysis:
• anthropological,
• sectoral, territorial,
• inter-program (in connection with other state development programs).
This requires the heads of project offices of ministries / departments / departments of education to have an integral vision, cross-sectoral understanding of development tasks, strategizing ways to achieve goals, taking into account data analysis.
It should be noted that the methods of calculating the indicators of educational development programs are changing, focusing the calculation formulas on a person and the development opportunities of each. For example, the coverage of additional education for children and its availability were previously calculated according to the data on the occupancy rate of programs of additional education for children of different directions, at present - according to the data on the participation of each child in programs of additional education, which represent opportunities for the development of the human potential of each.
Thus, the ideology of human orientation, territorial socio-economic integration of education development programs into the socio-economic development of regions and the country, impact investment - changes the technologies for designing data architecture, data analysis, and, consequently, changes the requirements for the competencies of educational leaders, their managerial thinking. and activities (see the description of the structure of the set of open data [12], information on the achievement of planned values of indicators (indicators) for government programs [13].
Changing methodological approaches to the development and implementation of state programs (education, including), change the methodology of monitoring in education. The role of monitoring changes from ascertaining (fixing data) to risk-oriented and motivating [14].
The new methodology changes the structure of projects that make up the education development program [15, 16, 17], and the assessment of their effectiveness in connection with the new design of the education development data architecture [18].
In this regard, many regions are beginning to change the models of training and certification of management personnel in the field of education [19, 20].
One of the backbone competencies in management is the competence of analyzing data on the development of education. For example, the attestation certificate of the director of a Moscow school includes indicators reflecting the competence of the director to work with data analytics and organize education development projects based on data analysis: a) data on the effectiveness of resource management, b) data on the contribution to quality education, c) data on development teaching staff, d) data on the conditions for providing education, e) data of public assessment of activities [21].
It is not only the competence of management that changes, the style of making management decisions in general changes: from situational to conceptual.
The task of the current stage is to ensure a competence-based transition from big data to smart big data: from the accumulation of an array of educational data - to the smart use of data for evidence-based management of development projects.
The analysis of public reports in the context of the application of data for evidence-based management and evidence-based policy was carried out by us on the basis of research questions:
a) how is the evidentiary discourse of public reports changing ?,
b) what types of data are used in public reports?
c) what digital services and data analysis methods - smart big data - are used in Public Reports?
Using the method of discourse analysis, the authors studied the application of the concept of "evidence-based management", "evidence-based politics" in the discourse of public reports. In the analysis, the species range of concepts was differentiated: frame (statistical) and scenario (dynamic).
As a result of the analysis, the following conclusions were formulated:
1) the discourse of public reports is changing towards the development of evidence-based education management based on educational data analytics.
2) the concepts of "evidence-based management", "evidence-based policy" are used in Public Reports, mainly not as a "scenario", but as a "frame" - a descriptive data frame, stating data in reports, but not using data to predict regional development scenarios, municipal education systems.
3) public reports reflect those educational data that are regulated by the requirements for the mandatory part of public reports, as well as those data, the possibility
the use of which is associated with the availability for the region, the municipality of a digital platform, analytics service and statistics of educational data.
The style of making management decisions in the context of data analytics was studied during the use of digital platforms and data analytics services by program participants.
The following methods were applied: a) a structured interview on the use of data sources and digital data analysis services to form management decisions; b) Alan Row's methodology (cognitive-situational model) for defining management decision-making styles. The classification of decision-making styles was taken as a basis, based on the criteria: 1) the cognitive complexity of the tasks being solved, 2) an orientation towards the way to achieve the goal. The criterion "cognitive complexity of the tasks to be solved" characterizes the complexity of the analyzed control object, the system of data sources about the controlled object, the complex of applied data analysis methods. The criterion "focus on the way to achieve the goal" characterizes the priorities of the leader, namely: focus on solving a problem or on people.
Figure 1 shows the criterion characteristics of the styles of management decisions with a highlighted analytical style, which we considered as the dominant of evidence-based management of educational systems.
Since Alan Rowe's methodology for studying the style of managerial decisions is based on two coordinates: cognitive complexity (preference for structure or uncertainty) and preferential orientation (toward people or tasks) [14], it allows us to identify two fundamentally opposite styles of forming managerial decisions - “directive- behavioral "and" conceptual-analytical "(Fig. 1). The styles of managerial decision-making are changing: from directive-behavioral to conceptual-analytical - in the context of the development of data analysis for the formation of managerial decisions and evidence-based development of education.
Fig.1. Essence of Management decision making styles (Alan Rowe)
Measurements according to the Alan Rowe method were carried out within the framework of the implementation of the program of additional professional education "Data analytics in public administration" twice: at the beginning of the development of the program and after its completion.
The program was originally designed in the structure of educational modules that develop the ability to use the technological data infrastructure, choose digital platforms and data analytics services that are appropriate for management tasks, the ability to identify and apply appropriate data analysis methods to various objects of management analytics, and the ability to form management decisions. This made it possible to study the dynamics of management decision-making styles built into the program.
The markers of management decision-making styles built into the methodology made it possible not only to see how immersion in evidence-based management affects the reflection of managers, but also to observe individual external circumstances of the choice of style, such as the level of managerial responsibility, work experience in the position, age, general managerial experience, professional installations.
3. Conclusions and recommendations
Based on research on management practices based on data, a competency model has been identified.
The structure of data-competence, characterizing the subject managing the educational system, as implementing the strategies of evidence-based development of such systems:
- Reflected professional attitude towards the management of education development projects based on data in the conditions of "open-data", "open-governance";
- the formation of basic concepts of data-based management methodology; knowledge of regulatory and legal regulators of assessment and monitoring in education;
- knowledge and application of information systems for assessing the quality of education; a comprehensive understanding of analytical services and systems of federal, regional and institutional significance;
- readiness and ability to form, at the imputed level of official responsibility, teams of "data-analysts" as subjects of analytical support for project management and education development programs;
- willingness and ability to develop a culture of data analysis and evidence-based management in a professional environment and organize the necessary measures to consolidate this culture in ethical standards / regulations of education management;
- the willingness and ability to respond in a mobile way to evidence based on the study of public opinion and evidence supported by scientific research in education.
For the formation of the above competencies, specialized programs of higher [24] additional professional education are required, as well as tools for assessing and certifying qualifications, data analytics standards for evidence-based project management and evidence-based policy.
In the management of public education development programs, evidence-based management methodology is integrated with project management methodology at all levels. At the strategic management level: strategic performance indicators, infrastructure and data sources are defined. At the level of project management activities, sources of alternative data analysis are determined based on feedback from citizens (public policy). At the level of project life cycle management, it is determined which data sets will indicate the dynamics of value development and the effectiveness of the project system in state development programs. At every level of evidence-based project management, this requires data competencies from leaders.
Currently, the scope of application of data-competencies in education management is expanding in connection with the integration of industry analytics with the National Data Management System [25], conceptually and technologically integrating all levels of analytics.
The methodology of evidence-based management is integrated with the methodology of project management at all levels. At the level of strategic management: strategic performance indicators, infrastructure and data sources are determined. At the level of project management activities, sources of alternative data analysis based on feedback from citizens (public policy) are determined. At the level of project lifecycle management, it is determined: which data sets will indicate the dynamics of value development and the effectiveness of the products of the project portfolio of state development programs. At every level of evidence-based project management, this requires data-competencies from managers.
Data-competencies are the basis in the structure of project management competence. The methodology of evidence-based management is integrated with the methodology of project management at the methodological level - project lifecycle management-defining: how the analysis of data sets in the project portfolio determines the configuration of changes at the strategic level and the level of project management activities through the evidence-based public policy.
4.Acknowledgments
The research was prepared with the grant “Methodology for the analysis of bulk data in education and its integration into training programs for teachers and heads of educational institutions in the logic “Pedagogy based on data,” “Management of education based on data” No. 19-29-14016 from Russian Foundation for Basic
Research in the competition for the best projects of interdisciplinary fundamental research “Fundamental scientific support of the general education digitalization processes.”
References
1. Decree of the Government of the Russian Federation of December 26, 2017 No. 1642 On the transfer of the state program "Development of Education" to project management from 2018. URL: http://government.ru/docs/30832/
2. Decree of the Government of the Russian Federation of May 26, 2021. No. 786 "Regulations on the management system of state programs of the Russian Federation." URL: http://government.ru/docs/all/134475/
3. Portal of state programs of the Russian Federation. URL: https://programs.gov.ru/
4. Levels of certification / Center for assessment and development of project management. URL: https://www.isopm.ru/sertifikatsiya/urovni/
5. Project Olympus / Annual competition for professional project management. URL: http://olimp.ac.gov.ru
6. GOST R ISO / IEC 20546-2021 “Information technology. Big data. Review and Dictionary ". Federal Agency for Technical Regulation and Metrology. URL: http://protect.gost.ru/document.aspx?control=7&id=240981
7. Project, program and portfolio management - Context and concepts. ISO 21500: 2021. URL: https://www.iso.org/ru/standard/75704.html
8.GOST R 54869-2011. National standard of the Russian Federation. Project management. Requirements for project management.
9. Impact Reporting & Investment Standards. Motreal, Quebec, Canada. 6-30. (2011)
10. Fiofanova O.A. Big data analysis in education: methodology and technologies. M: Delo, RANEPA, 2020.
11. Fiofanova O.A. The problem of integrating digital data analytics services: the teacher's competence in working with educational data. Moscow University Bulletin, Series 20, Pedagogical Education. No. 9, 2020.
12. Description of the structure of the open data set / Versions of the structure of the open data set / https://programs.gov.ru/opendata/7710349494-consolidatedreport2020/
13. Consolidated reports: information on the achievement of planned values of indicators (indicators) under state programs // https://programs.gov.ru/Portal/opendata
14. Motivating monitoring: The Ministry of Education will increase the responsibility of the regions for the implementation of national projects. URL: https://edu.ru/news/nacionalnyy-proekt-obrazovanie/motiviruyushchiy-monitoring-minprosveshcheniya-pov/
15. State program of the city of Moscow "Development of education in the city of Moscow (" Capital education "). URL: https://docs.cntd.ru/document/537907408
16. State program of the Sverdlovsk region "Development of the education system and implementation of youth policy in the Sverdlovsk region until 2025" (as amended on April 15, 2021) // https://docs.cntd.ru/document/561648866
17. Public declaration of the state program of the Khanty-Mansiysk Autonomous Okrug-Yugra "Development of Education" (Yugra 2020-2030) // https://depobr-molod.admhmao.ru/gosudarstvennye-programmy/gosudarstvennaya-programma-khanty-mansiyskogo-avtonomnogo -okruga-yugry-razvitie-obrazovaniya-vstupae / 2251775 / prezentatsiya-gosudarstvennoy-programmy-razvitie-obrazovaniya-vstupaet-v-silu-s-2019-goda /
18. The procedure for the development, implementation and evaluation of the effectiveness of state programs of the Perm Territory was approved by the Resolution of the Government of the Perm Territory dated November 21, 2019 No. 835-p "On approval of the Procedure for the development, implementation and evaluation of the effectiveness of state programs of the Perm Territory."
19. The new corporate university will solve the problems of metropolitan education / Vesti obrazovaniya // https://vogazeta.ru/articles/2021/7/2/city_education/17515-novyy_korporativnyy_universitet_budet_reshat_zadachi_stolichnogo_obrazovaniya
20. Certification of managers and candidates for compliance with the position of the head of a state educational organization. URL: https://www.mos.ru/donm/function/activities/attestatciia-rukovoditelei-i-kandidatov-na-sootvetstvie-dolzhnosti-rukovoditelia-gosudarstvennoi/
21. Education of Moscow: mechanisms, resources, results. A guide to open resources DONM. Moscow: 2020.
22. Fiofanova O.A .: Smart Big Data in public reports / Educational policy, 4 (84), 2020. URL: https://edpolicy.ru/smart-big-data
23. Rowe J., Boulgarides D. Decision Styles - A Perspective. - Leadership & Organization Development Journal, Vol. 4 No. 4, pp. 3-9. 1983.
24. The project of the magistracy of the IOM RANEPA was supported by the Vladimir Potanin Foundation. URL: https://www.ranepa.ru/sobytiya/novosti/proekt-magistratury-iom-rankhigs-podderzhan-fondom-vladimira-potanina/
25. National Data Management System / Analytical Center under the Government of the Russian Federation. URL: https://ac.gov.ru/projects/project/nacionalnaa-sistema-upravlenia-dannymi-nsud-41 (2020).
[1] Project, programme and portfolio management - Context and concepts. ISO 21500:2021. URL: https://www.iso.org/ru/standard/75704.html (2021)
[2] Impact Reporting & Investment Standards. Motreal, Quebec, Canada. 6-30. (2011)
National Data Management System. Analytical Center under the Government of the Russian Federation. URL: https://ac.gov.ru/projects/project/nacionalnaa-sistema-upravlenia-dannymi-nsud-41 (2020)
[3] Polkovnikov A.V., Dubovik M. F. Project management. 552p. – Moscow, Olymp-Business (2015)
[4] National Data Management System. Analytical Center under the Government of the Russian Federation. URL: https://ac.gov.ru/projects/project/nacionalnaa-sistema-upravlenia-dannymi-nsud-41 (2020)
[5] Fiofanova O.A.: Smart Big Data in public reports. Educational Policy. 4 (84), 70-78. URL: https://edpolicy.ru/smart-big-data (2020)
[6] Rowe J., Boulgarides D. Decision Styles - A Perspective. - Leadership & Organization Development Journal, Vol. 4 No. 4, pp. 3-9 (1983)
https://ru.ipma2021.world/program?tab=program&day=3
