Mauro Martini
Why
Evaluate?
ln 1993, The United States Government launched the “Reinventing Government”
policy, which has deeply influenced Public Administration practice all over
the world. However, contrary to what people may imagine, the “Reinventing
Government” theory does not provide for the State to be regarded as a “business”,
especially in view of the significant differences existing between the public
and private sectors; on the other hand, Public Administration is requested
to adopt a more entrepreneurial approach (Osborne and Gaebler). In particular,
the “Reinvention” suggests a more adequate distribution of resources, also
through benchmarking research among administration bodies, but also by implementing
projects aimed at improving administrative activity efficiency, effectiveness
and cost control. Nevertheless it is important to remember that the “Reinvention”
makes it necessary to focus attention and efforts on results (efficiency and
effectiveness controls), rather than on rules (legitimacy controls). In Italy,
the phase of legislative history displaying unequivocal elements and indications
as to the legislator’s intent to change Public Administration, by introducing
evaluative practices of some kind, dates back to the early ‘90s, through a
number of measures which ultimately also involved the Public Health sector.
The legislator’s intent, which is now expressed through a substantial number
of regulations (which by the way are often subject to review), risks being
thwarted if the contents of such rules is not be correctly acknowledged, first
of all by public administration managers and, all the more so, by the Managers
of the National Health Service. A complete understanding of the manager evaluation
processes therefore (necessarily) involves a correct interpretation of the
role managers are entrusted with as part of the complex “Reinventing Government”
process. Only once we have clarified and understood the principles regulating
“Reinvention” will it be easier to understand what a manager is expected to
do today. In such a context, evaluation can, through a feedback process, change
into a major tool for improvement: this is the foundation on which “change”
can be effectively built and implemented. The desire to be evaluated is instinctive,
although the existence of some sort of conflict between such a desire and
the fear of being evaluated is undeniable (Mohrman and Coll.). This motivation
(in both its components: desire and fear) becomes even stronger when the performance
is related to any kind of gratification, which need not be of an economic
nature. Despite the progressive introduction of management evaluation into
the Italian legislation covering public administration bodies, the opposition
to a concrete and effective implementation stage is chiefly brought about
by this inner conflict. The issue is therefore chiefly of a cultural nature,
within a context which should mainly involve the political and strategic management
teams of public administration bodies, which often appear to be the ones which
have greater distrust in the potentials of evaluation in general, but above
all when related to the performance of managers. In this regard, Fertonani
believes by right that “before asking department heads and team-mates to believe
in the importance of evaluation, it is Company Management and Staff Management
who should be satisfied of the importance of such methodologies”. This climate
of distrust is also fuelled by the lack of “basically objective, homogenous,
transparent and shared evaluation processes”, which means that it is essential
and absolutely urgent to create a “permanent staff performance evaluation
system that is transparent, in other words agreed with trade union representatives,
and also well-known and homogenous, and therefore equal for all staff members”
(Migliozzi). Also according to Ruffini: “the most critical issue of the evaluation
process is represented by the difficulty in making it objective”, which is
particularly true in the evaluation of executive staff. The need for “objectivity”
in evaluation criteria also results from the distinctive exceeding self-esteem
characterising most individuals, and hence the opinion that one’s activity
has been carried out in line with (if not beyond) the quality and quantity
standards required by the strategic Management. In fact, the Gaussian distribution
of staff performance says that only half of the values can be regarded to
be above average (Mohrman)!
Regulations
Governing the Evaluation of Managers
We are here briefly mentioning the chief reference laws regulating the evaluation
of National Health Service executive staff, which can be traced back to the
“Healthcare reform” decrees (Lgs D .no. 502/1992 and subsequent alteration
and additions) and to the 1998 – 2001 edition of the National Collective Labour
Agreement (CCNL), which also refers to certain major provisions of the Lgs
D. no. 29/1993 and of the Lgs D. no. 286/1999. We can divide such regulations
into three categories:
a) Provisions enunciating managers’ duties;
b) Provisions regulating evaluation procedures;
c) Provisions regulating the sanction application system.
a) Provisions Enunciating Managers’ Duties Lgs.
D. no. 502/1992 and Subsequent Alterations and Additions. Art. no. 15, Commas 1 and 2
1.… National Collective Labour Agreements provide for … general criteria for the graduation of managerial functions and for the assignment, evaluation and review of management assignments as well as for the attribution of the related additional pay depending on the functions assigned and on the related responsibilities for results.
2. Healthcare management is regulated by the legislative decree no. 29, of 3 February 1993, and subsequent alterations, save for the provisions of this decree …
[In this regard, the reader is reminded that the Lgs. D. no. 29/1993 – one of the cornerstones of the “Reinventing Government” in Italian legislation – deals with the “Rationalisation of organisation in public administration bodies and with the amendment of the regulations covering state employees”].
Lgs. D. no. 286/1999. Art. no. 5, Commas 1 and 2
1. Also on the basis of management control results, public administration bodies evaluate, in line with the applicable prescriptions of the National Collective Labour Agreements, the performance of their managers, as well as the behaviours relating to the development of professional, human and organisation resources they are entrusted with (organisational skills).
2. The evaluation of managers’ performance and organisational skills takes into special account the results of administrative and management activity …
[ Il Lgs. D. no. 286/1999, quoted under art. 26 of the National Collective Labour Agreement, NHS management area, 1998 – 2001, concerns the “Reorganisation and strengthening of the mechanisms and tools for monitoring and evaluating costs, outcome and results of the activity carried out by public administration bodies”; within the Italian legislation setting, this may be regarded as the most important directive relating to evaluation procedures].
b) Provisions Regulating Evaluation Procedures
Lgs. D. no. 502/1992 and Subsequent Alterations and Additions. Art. no. 15, Comma 5
The manager is subject to a triennial review; a manager with a facility assignment, whether simple or complex, is also subject to review at the end of the assignment. The reviews relate to the professional activities carried out and the results achieved and are performed by a technical committee, appointed by the general manager and headed by the department head. The positive outcome of the review is a prerequisite for the assignment or confirmation of professional or managerial tasks of greater importance.
Lgs. D. no. 502/1992 and Subsequent Alterations and Additions. Art. no. 15, Comma 6
In addition to the functions associated with their specific professional responsibilities, executives with a complex facility management assignment are also entrusted with facility management and organisation functions, to be carried out within the operating and management policy of the department they belong to; this is also accomplished through directives to all the staff employed in the facility and the adoption of the applicable decisions required to ensure proper fulfilment of the service and the adequacy of interventions with preventive, diagnostic, therapeutic and rehabilitative purposes, carried out in the facility they are responsible for. The manager is responsible for the effective and efficient management of the resources he is granted. The results of his management are subject to annual review through the Evaluation Nucleus.
National Collective Labour Agreement 1998 – 2001. Art. no. 31, Commas 1, 2, 3:
1. The organs in charge of the performance appraisal of managers (in the National Collective Labour Agreement healthcare, technical, administrative and professional area, the words “with a healthcare role” are added), pursuant to art. 15, commas 5 and 6 of the Lgs. D. no. 502/1992 are:
a) the Technical Committee;
b) the Evaluation Nucleus;
2. The body mentioned under comma 1, letter a) carries out the review:
a) of the professional activities performed and of the results achieved by all the managers, irrespective of their assignment, on a three-year basis;
b) of the managers who have a complex or simple facility management assignment, upon completion of the assignment itself;
c) of newly appointed managers, with the purpose of confirming their assignment, at the end of the first five-year period of service.
3. The body mentioned under comma 1, letter b) carries out the annual review: a) of the management results of complex facility managers and also of simple facility managers if they are responsible for resources management;
b) of the results achieved by all managers, including those listed under letter a), as they compare to their goals.
National Collective Labour Agreement 1998 – 2001. Art. no. 32, Commas 6, 7
6. The annual evaluation of managers by the Evaluation Nucleus has the objectives indicated under art. 31, comma 3, letters a) and b).
7. To merely provide an indication, the evaluation mentioned under comma 6 for complex or simple facility managers should – if the applicable conditions apply – relate to the management of the budget they are responsible for and to the human and instrumental resources they are actually granted, as well as all the functions delegated pursuant to the corporate deed and the evaluation of the organisation models adopted for the achievement of the goals, whereas for the other managers it concerns the compliance with performance objectives entrusted and the commitment and availability shown in working hour organisation versus goal achievement.
Lgs. D. no. 286/1999. Art. no. 5, Comma 2
… The evaluation is carried out on an annual basis. The evaluation procedure is inspired by the following principles: direct knowledge of the activity of the person subject to evaluation by the first-instance proponent or evaluator body, the approval or review of the evaluation by the second-instance competent or evaluator body and participation in the procedure by the person subject to evaluation.
c) Provisions Regulating the Sanction Application System
Lgs. D. no. 502/1992 and Subsequent Alterations and Additions Art. no. 15 – ter, Comma 3
The assignments covered by commas 1 and 2 are annulled, pursuant to the procedures provided for by the regulations in force and by the national collective labour agreements, in the event of: non compliance with the instructions given by the general or department management; failure to achieve the goals assigned; serious or reiterated liability; in all other cases provided for by the labour contracts. In the most serious cases, the general manager may back out of the employment contract, in line with the provisions of the civil code and of the national collective labour agreements.
Lgs. D. no. 29/1993. Art. no. 21, Comma 2
In the event of serious non-compliance with the instructions given by the competent body or of reiterated negative evaluation … the manager, subject to previous non-compliance notification and discussion, may be excluded from the conferment of further assignments within the same managerial level as the annulled assignment, for a period of not less than two years. In the most serious cases, the administration may back out of the employment contract, pursuant to the provisions of the civil code and of the collective agreements …
National Collective Labour Agreement 1998 – 2001. Art. no. 34
This provides for the
following steps:
a) Discussion with the manager, in the event of negative evaluation, with
the assistance of a trusted person;
b) Adoption of measures that are proportionate to both the position and the
extent of the non-compliance;
c) Typology of measures:
- partial or total loss of the result-related remuneration;
- assignment of a different task on a similar organisational level;
- assignment of a different task on a lower organisational level;
- termination of employment. In view of the complexity of the measures referred
to under commas 3 to 7 of art. 34, a summary table has been prepared (Tab.
3).
Lgs. D. no. 286/1999. Art. no. 5, Comma 4
The evaluation procedure described under comma 3, represents a premise for the application of the measures covered by article 21, commas 1 and 2, of the decree no. 29, on the subject of managerial responsibilities. In particular, the measures covered by comma 1 of the aforementioned article apply when the negative results of the administrative and management activity or the failure to achieve the objectives emerge from the ordinary annual evaluation procedures. However, when the serious risk of a negative result materialises earlier than the annual performance appraisal deadline, the evaluation procedure may be closed ahead of time. The evaluation procedure is also closed ahead of time in the cases provided for by comma 2 of the aforementioned article 21, of the decree no. 29.
A rather significant aspect is represented by the duplicity of the institutional competence involved in executives’ evaluation: the Evaluation Nucleus and the Technical Committee (keeping in mind that the body responsible for the switch of top executives from the outdated employment relationship to the new term-related appointment contract, headed by the Health Unit Manager, has only acquired temporary legal status, for the time required to enforce the provisions of the Lgs. D. no. 229/1999 and of the Lgs. D. no. 49/2000). In our opinion, the two bodies can only be actually distinguished because of their makeup (which is absolutely free for the Evaluation Nucleus, whereas the Technical Committee can only be chaired by the Department Head) and for the different periodicity of the reviews (which are annual for the Evaluation Nucleus, and triennial or upon assignment completion for the Technical Committee). Indeed, we find that any disquisition as to alleged discordances in the evaluation “contents”, detected by some, are ineffective, since they appear in fact difficult to differentiate, at least in substance. The objectives of the evaluation are therefore contained in the two aforementioned provisions (Lgs. D. no. 502/1992 and subsequent alterations and addition, and the National Collective Labour Agreement 1998 – 2001): a) For the Evaluation Nucleus the provisions refer to the “management results achieved”; the management is to prove “effective and efficient in line with resources and objectives”; art. 32 of the National Collective Labour Agreement also supplies significant specifications; b) For the Technical Committee, on the other hand, provisions refer to “professional activities” and “results achieved”. In our opinion no substantial differences exist, for the following reasons:
1. This definitions
ought to be reinterpreted in the light of all the regulations (including,
but not limited to, the Lgs. D. no. 502/1992 and the National Collective Labour
Agreement) regulating managerial duties;
2. Breaking down a manager’s professional profile in order to allocate to
different bodies the evaluation of the resulting “competences” appears to
be an illogic and what’s most meaningless procedure, also in view of the fact
that it would make no sense for the manager in question to achieve for instance
a positive assessment for his organisation skills and a negative one for budget
management, since it is in any case necessary to come to an overall final
evaluation as to the functional position covered;
3. Aside from the expression “results achieved”, which is used for both typologies,
the term “professional activities” (evaluation covered by the Technical Committee)
is absolutely generic, with an exegesis which appears to differentiate the
dynamic-modality element (activity, therefore actions and management modalities),
required to pursue management results (static–teleological element). Therefore,
the parallel existence of two performance assessment organs (the Evaluation
Nucleus and the Technical Committee) may only be logically justified by the
requirement to evaluate the executives’ performance from two different perspectives
and with diachronic modalities: one on a strictly annual basis and the other
one according to specific time intervals. Nevertheless, in such a context,
a close cooperation between the two bodies should anyhow be demanded and encouraged,
to make evaluations more effective.
Principles
and Subject of Evaluation
The regulations published in Italy over the last decade in connection with
the reform of Public Administration have gradually introduced rules and regulations
concerning management evaluation, without however going to the heart of the
methodology - except for general principles – leaving the solution to the
discretion of the Body or Enterprise. This may involve disadvantages, but
also advantages. It certainly reveals a lack of homogeneity, especially in
the contexts in which managers control absolutely identical outputs (in this
regard, the NHS represents the most significant example). The basic principles
of the evaluation methodology, quoted in the 1998 – 2001 edition of the National
Collective Labour Agreement for the National Health Service management area
(art. 32, comma 4), are basically three: the transparency principle, the information
and participation principle and the knowledge principle. Zerilli effectively
summarises such principles, by announcing that “The evaluation … is a right
for the individual himself, who, the larger the organisation he works in,
the less is he able to directly perceive how the organisation judges him and
what is expected of him. The individual … has first of all the right to be
judged objectively and according to a precise procedure and definite criteria;
secondly, he is entitled to know this judgement and eventually speak up for
himself”. Again in connection with the general principles of evaluation, we
ought to recall art. 5 of the Lgs. D. no. 286/1999, where comma 2 prescribes
“The evaluation procedure is prompted by the principles of direct knowledge
of the activity of the person subject to evaluation by the first-instance
proponent or evaluating organ” and “of participation of the evaluated subject
to the procedure”. The transparency procedure is a typical feature of the
evaluation methodology: all the procedure phases, in terms of modalities,
implementation times and criteria employed, must be known beforehand by the
strategic Management, by all the managers and by the related trade union representatives.
Transparency applies in any case to general methodology, since, on the other
hand evaluation results must be exclusively made known to the person concerned
(also in compliance with the provisions of the law no. 241/1990) and the Enterprise
Management (in this regard, art. 6, comma 2, of the Lgs. D. no. 286/1999 recalls
that the bodies in charge of the evaluation “confidentially report to the
policy bodies”). The information principle takes shape according to three
modalities: a) communication as to the implementation of the procedure (this
means the person concerned is to be informed that an evaluation procedure
that concerns him has started); b) participation in the procedure by the individual
subject to evaluation, possibly also through discussion, especially in the
event of negative evaluation; c) final results of the evaluation, for which
the Enterprise Management is responsible. The third principle relates to the
direct acquaintance with the individual subject to evaluation by the evaluator
who “in the first instance prepares the evaluation proposal based on which
the assessment body is called to take a stand.”“ (please refer to art. 32
of the National Collective Labour Agreement).
As regards the subject of the evaluation of NHS managers, we hereby refer to the contents of art. 32 of the National Collective Labour Agreement, as well as of art. 5 of the Lgs. D. no. 286/1999 (despite the discretionary power involved in its application), listed in the table 5. As regards the chronology of evaluation, in addition to the annual, triennial and assignment completion deadlines, relating to the specific regulation and contractual indications, it is worthwhile mentioning once again (especially in the absence of a similar reference regulation) art. 5 of the Lgs. D. no. 286/1999, which prescribes under comma 4 that “when the serious risk of a negative result materialises prior to the annual deadline, the evaluation procedure may be closed in advance”, moreover assessing that “the evaluation procedure will be closed in advance … in the cases provided for by comma 2, of the above-mentioned article 21, of the decree no. 29>.
[The reader is reminded, in this regard, that the Lgs. D. no. 29/1993, under art. 21, comma 2, prescribes that “In the event of serious non-compliance with the directives given by the competent body or of a specific responsibility for the negative results of the administrative activity and management, the manager, subject to previous non-compliance notification and discussion, may be excluded from the conferment of further assignments within the same managerial level as the annulled assignment, for a period of not less than two years. In the most serious cases, the administration may back out of the employment contract, pursuant to the provisions of the civil code and of the collective agreements”].
Who
Evaluates?
The regulations previously recalled clearly refer to the bodies responsible
for the evaluation, depending on the case in point; however, both in order
to comply with the contractual principle relating to “direct acquaintance”
with the manager subject to evaluation, and because evaluators do not always
possess all the useful and/or necessary information, the need for other professional
figures to take part in the procedure is unquestionable, starting of course
form the manager’s immediate superior, but also the latter’s superior, etc.
The chief methods suggested in literature are the “one
over one” evaluations (only the direct superior evaluates; however, irrespective
of the regulations in force, this hypothesis appears inappropriate), “multiple
appraisal” (the method which is certainly the most advisable) and “multisource
feedback” (not actionable, in Public Health, for obvious reasons). In
the Ferrara ASL (Local Health Enterprise) experience, the Evaluation Nucleus
employs a “multiple appraisal” method, by entrusting the Direct Superior with
the first approach. The contribution by the institutional body only takes
place at a further stage, with the acquisition, if necessary, of any further
useful information (evaluation by the manager the direct superior reports
to, documentation also from other company departments, etc.), especially if
critical issues have emerged.
The
Evaluation Scales
One last question relates to evaluation scales, which evaluators may use to
define the manager’s performance in terms of quality and quantity (for instance:
“insufficient”, “sufficient” and “excellent” performance). In this regard,
Migliozzi (rightly, in our opinion) advises against the use of scales that
either have an excessive number of grades (owing to the difficulty in distinguishing
between two subsequent measurements), or have a too limited number of grades
(owing to the difficulties that are likely to be experienced in the choice
between two subsequent measurements). Nevertheless there is a legitimate doubt
as to the use of odd scales, which involve the disadvantage of encouraging
evaluators to choose “diplomatic” average values, or of even scales, which
on the other hand involve a choice between a positive value and a negative
one. According to Fertonani “If there is a high number, it often proves rather
difficult to identify the difference between one and the other; if, on the
other hand, the number is low , it may become embarrassing for the evaluator
to carry out a choice involving judgements that are too “tranchants”. As a
rule, one does not drop below three , if there are more than seven grades
… If they are odd … there can be a tendency towards average judgements … if,
on the other hand, they are even, the evaluator is forced always to choose
between a more positive and a more negative grade, and the evaluation can
be therefore distorted”.
The
Evaluation Protocol: An Example
The Evaluation Nucleus of the Ferrara Health Enterprise – in compliance with
the above mentioned principles (transparency, information and knowledge/acquaintance)
– has defined evaluation protocols in the management area, in agreement with
both the trade union representatives and the Enterprise Managers. The protocol
is presented in a concise manner, and includes the parts which are of greater
interest for explanation purposes (on the other hand, the original, resolved
by the Management, also contains the chief legal and authority references).
A necessary condition for the presentation of managers’ evaluation protocols
is the warning not to deal with the contents in an excessively strict manner.
The evaluation is a process that must flexibly fit in with the context: if
the evaluation culture is sufficiently widespread within the company framework,
it is possible to introduce subtleties and nuances that characterise individual
career briefs. Should this not be the case, at least during the early stages
of procedure implementation it is worthwhile to content oneself with the definition
of the items characterising homogenous macro-groups and set to tackle the
difficult task of convincing employees to “accept” the evaluation, encouraging
cooperative behaviours through preliminary meetings, seminars and/or any other
activity that proves fit to obtain full support from all managers, also because
many of them will be called in turn to personally take part in the evaluation
of their co-operators (the department head, for instance, for the complex
facility managers, etc.). Furthermore, the evaluation protocols (whose organisation
has to be implemented by experts on the subject) must subsequently and unavoidably
(also in compliance with the transparency, information and acquaintance principles)
be fully shared by the review bodies and the trade union representatives,
as well as by the Strategic Management, so as to be made known to all the
company managers. Nevertheless, one must always keep into account that the
chief aim of an evaluation activity is not that of arranging beforehand a
tool aimed at applying sanctions for the Strategic Management, but the improvement
of efficiency, of effectiveness and of the quality of service provided, also
trough reflections shared by the manager subject to evaluation and the evaluator,
especially in the event that the first stage of the evaluation is accomplished
with the cooperation of the direct superior.The protocols employed by the
Evaluation Nucleus of the Ferrara Health Enterprise (whose basic contents
are listed in the chapter) are agreed among the institutional stakeholders
before being resolved and becoming operational.
To organise the protocols, it has been thought essential, in order to define the items on which the evaluation forms are based, to first of all identify:
I. The management level
a) enterprise;
b) department;
c) facility;
d) towards clients;
e) towards stakeholders;
II. The reference
That is the management responsibility profiles subject to evaluation, which define the “items”;
III. The methodology
Relating
to the contents of the evaluation.
Pursuant
to the directives of art. 32 of the National Collective Labour Agreement 1998
– 2001, 10 items have therefore been identified: these must be regarded as
absolutely indicative and to be used especially during the implementation
of evaluation, and should subsequently be adapted to every context or, what
is even better, to every manager. The items involve an evaluation according
to a five-point scale (see below). As regards the annual evaluation, the procedure
employed in the Ferrara Heath Enterprise is the “multiple appraisal” type:
first of all, the evaluation form (see facsimile
below) is handed to the direct superior (or direct superiors, if applicable),
who fills in the form with the participation of the manger who is the subject
of the evaluation; the form is subsequently forwarded to the Evaluation Nucleus,
for the necessary integrations and conclusions, also with the use of other
offices, bodies or operating units, which, somehow, may contribute to the
supply of useful information, but of course also through annual reports concerning
the evaluation of the result objectives. In our opinion, the triennial and
assignment completion evaluations, peculiar to the Technical Committee, in
whatever manner accomplished (in our opinion, owing to the above mentioned
reasons, through similar items) must provide for the acquisition of periodic
reports of the Evaluation Nucleus and (in view of the significant lapse of
time involved by the review and in compliance with the participation principle)
also of a curriculum vitae of the manager, which will stimulate self-review
opportunities. As mentioned, a five-point evaluation scale has been employed,
in line with the strongest suggestions provided in literature and with the
prevailing trends of the organisations which have already made use of it.
In practice, for each item (see table
9), the evaluator is requested to express his opinion with a score which
does not represent a “mark”, but rather a qualitative attribute. The use of
a number, rather than of the applicable attribute, also allows a greater speed
in filling in the forms.
The sum of the scores assigned for each item on the form leads to an overall evaluation, which needs to be adequately interpreted (please refer to the hypotheses recalled in table 10, should the items be 10). In any case, it is worthwhile pointing out that, irrespective of the result achieved through the performance appraisal form, the procedure adopted by the Ferrara Enterprise Evaluation Nucleus is absolutely “observant of civil rights” for management staff. This is absolutely necessary, especially during the initial (and often long) evaluation system implementation phase, which has to be adequately known, both by the evaluators and by the evaluated subjects, and must in any case always be viewed as a tool for the improvement of the procedures and services supplied by the enterprise, rather than as a tool for applying sanctions. The referenced evaluation procedure is very effectively summarised by a flowchart, which also provides an indication as to final hypotheses.
(trad. Interpres sas - Giussano)
Mauro Martini
Direttore del Nucleo di Valutazione
dell'Azienda Sanitaria di Ferrara
