Selection committee guide

Other format:
PDF version (210 KB)

Foreword

This document is a guide for members of the Vanier Canada Graduate Scholarships (Vanier CGS) selection committees of the three federal granting agencies: the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC). It describes the review process for members and chairs of these selection committees, as well as the policies, guidelines and deliverables that define each stage of the review process.

Most members of the Vanier CGS selection committees are appointed from the Canadian academic community and are familiar with the mandate, structures and programs of the federal granting agencies. For new members and those appointed from other sectors, detailed information regarding the respective federal granting agency mandates can be found on the agencies' websites:

1. Overview of the Vanier Canada Graduate Scholarship program

The Government of Canada launched the Vanier CGS program in 2008 to strengthen Canada's ability to attract and retain world-class doctoral students and establish Canada as a global centre of excellence in research and higher learning. Vanier Scholars demonstrate leadership skills and a high standard of scholarly achievement in graduate studies in the social sciences and/or humanities, natural sciences and/or engineering and health.

Students wishing to apply for a Vanier CGS must do so through the Canadian institution to which they are applying for doctoral studies. Based on their quotas, Canadian institutions will forward a limited number of nominations to the Vanier CGS program.

Up to 167 Vanier Scholars are awarded each year and are equally divided among the three federal granting agencies.

The Vanier-Banting Secretariat, which is housed within the Canadian Institutes of Health Research, is responsible for the day-to-day administration of the program.

2. Roles and responsibilities

2.1 Selection committee members

Responsibilities of selection committee members include:

Specific responsibilities of assigned reviewers include:

Specific responsibilities of assigned readers include:

2.2 Selection committee chair

Selection committee chairs play an important oversight role on the committee and provide guidance to the Vanier-Banting Secretariat on the delivery of the program.

During the selection committee meeting, the selection committee chair ensures the orderly and complete evaluation of nominations and the transmission of recommendations to the Secretariat. The review process must ensure that all important aspects of a nomination are considered. The chair leads the selection committee's efforts to maintain the highest quality of evaluation and to ensure consistency and equity of approach during the selection committee meeting.

The chair will represent the selection committee in dealings with the Secretariat on policy issues. This includes submitting a report to the Secretariat following the selection committee meeting.

The chairs might be asked to make a presentation to the Tri-Agencies Programs (TAPs) steering committee on the competition results and the policy discussions, in person or by teleconference. More details will be provided should this be required.

The Secretariat asks selection committee members not to make formal recommendations for a future chair. Advice on potential chairs can be given to the Secretariat.

2.3 Tri-Agency Programs steering committee

The TAPs steering committee approves the recommendations put forward by the agency-specific selection committees.

The TAPs steering committee membership is based on role of the following five individuals:

Chair:

Members:

2.4 The Vanier-Banting Secretariat

The main responsibilities of the Secretariat are as follows:

The Secretariat staff are not selection committee members and do not have voting rights on the selection committee.

2.5 Executive director, manager and team leader

The main responsibilities of the Vanier CGS Executive director, Manager and team leader include:

Communication of secure documents

Throughout the competition cycle, the Secretariat staff may need to provide you with documents which contain sensitive information. In most cases, these documents will be shared by posting them on the appropriate pages within ResearchNet – the electronic web portal used for review submission.

3. The review process

The review process consists of the following stages, which are further described later in this document:

A complete nomination consists of:

Notes:

3.1. Prior to each agency-specific selection committee meeting

3.1.1. Briefing on the selection process and calibration sessions

These sessions are organized to provide an overview of the Vanier CGS program's review process and selection criteria. To ensure that all selection committee members assess nominations in a similar fashion, the main focus of these webinars/teleconferences is to hold a calibration session(s).

More information on the tri-agency briefing and calibration sessions is sent by email prior to the webinars/teleconferences.

3.1.2. Receipt and assignment of nominations

Nominations are received at the Vanier-Banting Secretariat and the program administrator assigns them to committee members.

The Vanier CGS agency-specific selection committees are multidisciplinary. Therefore, selection committee members are asked to review nominations in a number of different disciplines related to the federal granting agency's mandate, including nominations in disciplines that may not be their own. Each nomination is reviewed by two selection committee members. One selection committee member will serve as the primary reviewer and the other as the secondary reviewer.

Once nominations have been received at the Secretariat, they are assigned to selection committee members using ResearchNet - the electronic web portal used for review of nominations. The Secretariat staff will seek to balance workload, taking into consideration potential conflicts of interest, language capabilities and areas of expertise. Each selection committee member is responsible for their reviewer assignments.

As soon as the assignment of nominations is made, each committee member will be granted access to ResearchNet to review their Vanier Canada Graduate Scholarships assignments. Members will receive an email notification advising them that their assignments are available.

3.1.3. Identification of conflicts of interest by selection committee members

Please review the Federal Research Funding Organizations Policy on Conflict of Interest.

Each selection committee member is granted access to ResearchNet to review their assigned Vanier CGS nominations. Selection committee members will receive an e-mail notification advising them once their assigned nominations are available on-line.

Using ResearchNet, selection committee members are asked to identify any conflicts of interest in reviewing nominations to which they have been assigned. Should a selection committee member feel unable, for any reason, to review an assigned nomination, the selection committee member should contact the Secretariat staff immediately and a new reviewer will be re-assigned to the nomination as appropriate.

Are you in conflict of interest with the candidate?

The following guidelines governing conflicts of interest apply to the evaluation of Vanier Canada Graduate Scholarship nominations. Guidelines of this nature cannot foresee all possible situations, and the federal granting agencies rely on the judgment of committee members when identifying conflicts. A member may not act as a reviewer for an application if:

When a committee member is in conflict with a particular application, the member will be asked to leave the room before the deliberations on that application begins.

It is the responsibility of committee members to declare any conflicts of interest prior to the review of an application. In cases where the guidelines do not clearly describe a situation, or where the committee or chair has difficulties making a decision about a particular situation, Vanier-Banting Secretariat staff has the responsibility to rule.

3.1.4. Review and pre-scoring of nominations

Selection committee members review and pre-score their assigned nominations using ResearchNet.

  1. Introduction
  2. Using ResearchNet, selection committee members are to pre-score all nominations assigned to them and are to submit their pre-scores electronically to the Secretariat prior to the selection committee meeting. A specific deadline date will be communicated to committee members in due course.

    Is the application eligible for support?

    The Secretariat is responsible for screening all nominations against eligibility requirements to ensure that the candidate is eligible to apply to the Vanier CGS program. Selection committee members are invited to bring any concerns or questions regarding the eligibility of a candidate to the attention of the Secretariat staff, but should proceed with the review of the nomination while the situation is assessed.

    Is the proposed research contrary to the Tri-Agency Framework: Responsible Conduct of Research?

    If you judge that the proposed research may be contrary to the Tri-Agency Framework: Responsible Conduct of Research or if you have any ethical concerns with respect to the nomination, proceed with the review as usual and inform the Secretariat of your concerns as soon as possible.

  3. Selection criteria
  4. When scoring nominations, provide scores that establish the rank of the nomination relative to the other nominations in the cohort, keeping in mind the stage and nature of the candidate’s academic career. Nominations must receive a score of 3.1 or higher for each of the three criteria in order to be considered for funding.

    Candidates are evaluated and selected on the basis of the following three criteria, which are weighted equally by the selection committees:

    1. academic excellence;
    2. research potential; and
    3. leadership (potential and demonstrated ability).

    1. Academic excellence, as demonstrated by past academic results and by transcripts, awards and distinctions.

    The candidate's research history and the impact of their activities in their area of expertise to date are important indicators of their potential as research leaders of tomorrow. Reviewers should consider the sphere of influence of candidates relative to others along the following continuum of expanding impact:

    In evaluating this criterion, both the prestige of the Vanier CGS and the stage and nature of the candidate's academic career should be considered. In the case of health professionals, consideration should be given to standards of research productivity, etc. for their level of experience/qualifications relative to a PhD student. For applicants who have relevant work experience, scientific productivity prior to graduate school should be considered.

    Supporting evidence to be evaluated for this criterion:

    2. Research potential, as demonstrated by the candidates research history, his/her interest in discovery, the proposed research and its potential contribution to the advancement of knowledge in the field, the potential benefit to Canadians, and any anticipated outcomes.

    Supporting evidence to be evaluated for this criterion:

    3. Leadership (potential and demonstrated ability):

    Given the prestige of the Vanier CGS program, this is an important criterion that has to be assessed in an indirect manner, since there is no opportunity for the selection committee to interview candidates.

    Leadership could include the following:

This list is not intended to include all possible categories and is provided for guidance only.

Assessment of leadership

Leadership could be assessed with the following personal and/or social skills:

More information on leadership is available in the SSHRC-funded study, Leadership at the Graduate Studies and Postdoctoral Levels [ PDF (260 KB) ].

Supporting evidence to be evaluated for this criterion:

Pre-scoring

Selection committee members assign a score between 0.1 and 9.0 (in increments of 0.1, with 9.0 being the highest and 0.1 being lowest) to their assigned nominations for each of the three criteria. ResearchNet will automatically calculate the nomination's overall score by averaging the score assigned to each criterion.

In order to encourage selection committee members to differentiate between highly promising candidates and to ensure that the full range of the scale is used, a forced binning system has been implemented. The binning is done on the overall score and not on each criterion. The nominations assigned to a selection committee member must fall within each of the three following bins according to the proportion indicated in the table below:

Funding recommendation Score Proportion
Recommended 6.1 – 9.0 25%
Could be recommended 3.1 – 6.0 25%
Not recommended 0.1 – 3.0 25%

The remaining 25 % may be distributed between the three bins as selection committee members deem appropriate, in order to compensate for a relatively strong or a relatively weak subset of nominations. We recommend that selection committee members review a number of nominations before entering scores, to establish a frame of reference for selection committee members to score all of their assigned nominations. An electronic worksheet is provided to members to assist with the task of scoring and binning.

In order for this system to work effectively, it is essential that the entire range within a bin be used. Therefore, every effort should be made to delineate nominations within a bin. For example, if 3 nominations are placed within the top bin then one should be ranked highest and one lowest with the remaining nomination ranked in between, if appropriate.

As nomination volumes for this program are difficult to predict, reviewers are encouraged to scale the level of effort for reviews of lower quality nominations to the number of nomination assigned (the higher the nomination volume, the lower the time spent on nominations deemed to be in the "not recommended" range). Depending on the nomination volume received, variations on the forced binning system and/or triage of lower quality nominations may be implemented.

Note: Selection committee members are required to bring their personal notes on their assigned nominations to the selection committee meeting. All notes will be securely destroyed after the selection committee meeting.

Reviewers' pre-scores must be submitted electronically to the Secretariat using ResearchNet prior to the selection committee meeting. Selection committee members are encouraged to contact the Secretariat staff at any time if they need assistance during the pre-scoring process.

3.1.5. Preliminary ranking of nominations and assignment of readers

The Secretariat staff prepares a ranked list of nominations, provides selection committee members with a list of nominations to be discussed at the selection committee meeting and assigns a reader to each nomination to be discussed.

Once all the pre-scores have been received, the Secretariat staff will calculate the average of the two reviewer’s overall scores and rank the nominations from the highest to the lowest. The pre-score ranking of all nominations will be made available to selection committee members prior to the selection committee meeting.

Nominations being discussed at the selection committee meeting are assigned to a third selection committee member, who will have the role of reader (approximately five nominations per selection committee member). The reader is responsible for reading the nomination prior to the selection committee meeting and must be ready to provide comments on the nomination's strengths and weaknesses during selection committee deliberations.

3.2. During each agency-specific selection committee meeting

3.2.1. Review process during the selection committee meeting in Ottawa

Selection committee members meet in Ottawa for the Vanier CGS selection committee meeting. At the selection committee meeting, approximately 65 nominations (nominations ranked 31-70, discrepancies of 3.0 or higher and those identified by selection committee members) are discussed one at a time.

The primary reviewer assigned to a nomination is responsible for verbally summarizing the strengths and weaknesses of the nomination as well as their overall score as a lead-in to selection committee discussion. The secondary reviewer will provide comments to highlight agreement or discrepancies with the primary reviewer's analysis.  The reader will then be given the opportunity to add any additional information that they feels is required by the selection committee in order to assess the nomination. Comments or questions may be raised by the other selection committee members in order to clarify any of the information presented by the reviewers and reader.

Following the comments from the assigned reviewers/reader and the selection committee's discussion, the selection committee will come to a consensus on the nomination's final score. It is expected that the selection committee will decide on a consensus score for each of the three selection criterion and these will then be averaged to arrive at a final score for the nomination. To be considered eligible for funding, applicants must attain an average score of at least 3.1 in each of the three criteria.

The committee's final rating on each criterion for an individual nomination will be made available to the nominee following the competition. This will provide feedback to the nominee on the relative strengths and weaknesses of their nomination and its relative ranking. Committee members will not submit written comments on individual applications

The final ranked list of all applications will be submitted to the Tri-agency programs steering committee for approval.

Any problems encountered in the review of nominations during the selection committee meeting should be brought to the attention of the Secretariat staff.

3.2.2. Production of the final ranked lists

Selection committee members are invited to comment on the list. More particularly:

Each agency-specific Vanier CGS selection committee will forward a final ranked list to the TAP steering committee for final approval.

3.2.3. Selection Committee's policy discussion

The Secretariat will lead the selection committee's policy discussion.

At the end of the Vanier CGS selection committee meeting, once the ranked list is finalized, selection committee members are asked to participate in a policy discussion. The discussion generally includes comments on the quality of nominations reviewed, possible improvements to the review process, future selection committee membership needs, competition and travel logistics, hotel accommodations, the Secretariat's administrative services, policies, etc.

Note: Selection committee members are invited to give some thought to individuals who would be appropriate to serve as future selection committee members. Any suggestions can be provided to the Secretariat staff.

3.3 Following each agency-specific selection committee meeting

3.3.1. Preparation and submission of the chair's report

Following the meeting, each committee chair, in consultation with committee members and the Secretariat staff, prepares and submits the chair's report to the Secretariat.

This report will be presented to the TAPs steering committee and is a means of communicating the selection committee's recommendations, comments and suggestions for improvements to the competition process and program policies.

In addition to the final ranked list of all applications, the report should include, where relevant, comments on such matters as:

4. Policies and guidelines

In addition to the conflict of interest guidelines noted above, please keep in mind the following policies and guidelines when reviewing nominations.

4.1. Confidentiality of nomination material

When logged onto ResearchNet, selection committee members are asked to read and agree to follow the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers, which describes the expectations and requirements of the Tri-Agencies.

Nominations are provided to selection committee members in confidence and should be used for review purposes only. Such material should be kept in a secure place, not accessible to colleagues or students. In addition, material that the selection committee member no longer requires should be destroyed in a secure manner (i.e., shredded). All binders, files, personal notes, assessments, etc. must be left with the Secretariat staff at the conclusion of the selection committee meeting. If the Secretariat staff requires a selection committee member's assistance to provide additional information on a particular case after the competition, the selection committee member will be provided with new copies of the relevant material.

4.2. Confidentiality of recommendations

All funding recommendations are subject to approval by the TAPs steering committee and may be changed for reasons of budget, administrative error or lack of full compliance to the federal granting agency policies.

All matters discussed during selection committee meetings or teleconferences are confidential. Notifying candidates of the results of selection committee deliberations is the responsibility of the Secretariat, following the approval by the TAPs steering committee.

Discussions of all nominations are confidential and must not be divulged to others. Any release of information to a candidate must be done through the Secretariat.

Results must not be disclosed by selection committee members prior to the Secretariat official release. If approached by a candidate and/or institution representatives concerning a decision or any other matter, selection committee members should decline discussion and refer the individual to the Secretariat. Secretariat staff will act as a liaison between the Vanier CGS selection committee and the candidate/institution.

4.3. The Privacy Act

Canada's Privacy Act stipulates that personal information provided by candidates must be used only for the purpose of assessing nominations and making funding decisions. The use or disclosure of such information for any other purpose is illegal.

The information collected for this purpose must be directly from the individual. It may be collected from other sources only as part of the formal review process.

In view of the large number of nominations, selection committee members will not be required to provide written assessments of individual nominations. Candidates have a right to access information that is held by the Secretariat about their nomination. However, they do not have the right to access the names of those who reviewed their nomination.

A list of each agency-specific selection committee members will be published on the Vanier CGS website approximately 60 days after funding decisions have been announced.

It is important that selection committee members adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations.

4.4. The Canadian Human Rights Act

The activities of the federal granting agencies are subject to the Canadian Human Rights Act. The purpose of the Act is to give effect to the principle that every individual should have equal opportunity with other individuals to make the life that an individual is able and wishes to have, consistent with the duties and obligations as a member of society, without being hindered or prevented from doing so by discriminatory practices.

For all purposes of the Act, it is discriminatory practice to deny a service to an individual or to differentiate adversely in relation to any individual in the provision of that service based on that individual's race, national or ethnic origin, colour, religion, age, sex, marital status, family status, disability or conviction for an offence for which a pardon has been granted. Where the ground of discrimination is pregnancy or childbirth, the discrimination is deemed to be on the ground of sex.

4.5. Official Languages

The federal granting agencies, like all other federal institutions, have a key role to play in the implementation of the Official Languages Act. The Secretariat has an obligation to ensure that:

The federal granting agencies must ensure that their committees and their staff are fully aware of their obligations and rights regarding official languages by providing documentation on official languages to employees and members and by including relevant guidelines in the instructions to selection committees.

In accordance with their active offer of bilingual service to the public, the Secretariat will try to appoint as many experts as possible with the appropriate language capabilities to serve on selection committees.

Selection committees must ensure that all nominations receive a full and detailed evaluation (subject to the guidelines regarding level of effort for lower quality nominations as noted above), regardless of the official language of presentation. On occasion, this may entail consultation with Secretariat staff to identify Vanier CGS selection committee members with adequate linguistic capability.

4.6. Ethical considerations

Selection committee members play an important role in alerting staff to any concerns they may have with respect to the ethics of the research being proposed. Examples of problems include the following:

If a selection committee member has any concerns with respect to ethical matters, these should be discussed immediately with the Secretariat staff to determine if there is a means of resolving any apparent problems quickly or if the awarding of a scholarship should be delayed pending the resolution of the problem. The review of nominations should continue without being influences by this issue.

4.7. Responsible conduct of research

The three federal granting agencies have defined their expectations with respect to scientific integrity in the Tri-Agency Framework: Responsible Conduct of Research.

Research

The federal granting agencies expect the highest standard of integrity in the research they fund. The electronic submission of an application to the Secretariat commits the applicant to a number of principles, including compliance with the integrity policy. Should selection committee members identify, during the review process, what appears to be a lack of integrity they should bring it to the attention of Secretariat staff at the earliest opportunity. The Secretariat will refer any allegations to the Secretariat for the Responsible Conduct of Research for investigation. Such allegations should not be a consideration during the review process, nor should they be part of the committee's evaluation discussions.

A selection committee member may, during the review process, encounter possible misconduct situations (e.g., apparent misrepresentation of publications and/or data, plagiarism and other problems such as a lack of appropriate control/monitoring within the institution itself or undue restriction on the dissemination of research supported by federal funds). Selection committee members should alert the Secretariat staff of these situations at the earliest opportunity.

Selection and review process

The Secretariat expects the highest standards of integrity in the review process that it manages. The Tri-Agency Framework: Responsible Conduct of Research indicates that information provided by candidates for the purpose of selection and review cannot be used by reviewers without the author's permission. In addition, a reviewer should reveal to the federal granting agencies any material conflict of interest, financial or other, that might influence the Secretariat's decision on whether the reviewer should review nominations. Selection committee members are responsible for respecting the confidentiality of nomination material and for declaring material conflicts of interest. Should selection committee members become aware of a situation that violates the integrity of the review process, they should discuss this immediately with Secretariat staff.

5. Selection committee membership

5.1. Recruitment process

The federal granting agencies regularly solicit nominations for membership on their selection committees from institutions, industry and/or the government sector. Nominations are also put forward by previous selection committee members or by the Secretariat staff.

The membership slate for each selection committee is prepared by the Secretariat staff and is subject to final approval by the Manager of the Secretariat. Chosen in a similar way, the selection committee chair is a member who has already served on a selection committee for one or two competitions.

5.2. Selection committee member criteria

The most important criterion governing selection committee membership is academic and research excellence. The value of the review process rests on the credibility of the selection committee members and their recognized expertise and productivity in their field(s). Although each selection committee is representative of the community it serves, selection committee members themselves are not expected to act as representatives of any particular group, institution, region or country. Selection committees are structured to ensure:

5.3. Term of membership

The renewal of membership is essential to ensure the vitality of the selection committee process. Rotation of membership allows for broader representation of institutions and complementary expertise. However, continuity is also desirable, since the presence of experienced selection committee members on the selection committee promotes consistency in the review process and assists in the orientation of new selection committee members.

Selection committee members usually serve no more than three consecutive years. The extension of a term beyond three years for one additional year may be accepted in exceptional circumstances. Selection committee members who have completed their three-year term on the selection committee may be appointed as chairs for an additional two years.

Under normal circumstances, approximately one-third of committee members retire each year.

Date modified: