2016 Audit Quality Indicators Pilot Project Interim Report WHY A PILOT? To date, there has been limited information available to audit committees to measure and evaluate the quality of an audit. In response, there has been increased international activity around the use of audit quality indicators (AQIs), which are quantitative measures about the audit process. In 2016, CPAB launched an AQI Pilot project (Pilot) with six Canadian audit committees and their external auditors to get feedback about the usefulness of AQIs and to support the broader national and international discussions. CPAB’s Pilot is exploratory in nature and meant to encourage innovation in the way audit committees approach the use of AQIs. In particular, CPAB sought feedback on:
●
Whether AQIs provide additional transparency into the audit that is useful for the audit committee.
●
How to best leverage AQIs.
●
How AQIs can be integrated into regular audit committee discussions in an effective and efficient way for all parties in the audit process, including audit firms, management and audit committees.
This document summarizes the key learnings to date. Our observations suggest to us that AQIs could have significant potential to positively impact audit quality. We encourage audit committees, management and audit firms to continue to explore how AQIs can be integrated into their audit processes.
PILOT PROJECT DETAILS What we did We called for participants in February 2016 and worked with their external audit firm and management to determine the following:
●
Number and nature of AQIs of interest.
●
How best to integrate the AQIs into regular audit committee processes.
Reporting mechanisms and frequency.
●
2016 AQI Pilot Project
INTERIM REPORT
2016 AQI Pilot Participants Company
Audit Committee Chair
Air Canada
Chris Clark
Intact Financial Corporation
Eileen Mercier
Magna International Inc.
Lawrence Worrall
Royal Bank of Canada
David Denison
Sun Life Financial Inc.
William Anderson
Telus Corporation
William MacKinnon
1
Audit Quality Indicators Resources Public Company Accounting Oversight Board (PCAOB) Concept Release on Audit Quality Indicators Center for Audit Quality (CAQ) Approach to Audit Quality Indicators Guidance to Audit Committees on The Accounting and Corporate Regulatory Authority of Singapore’s (ACRA) Audit Quality Indicators Disclosure Framework
While CPAB provided Pilot participants with some guidance (see box left), no specific requirements were given. Participants were encouraged to determine how to work with AQIs in a way that best suited their goals and areas of interest. A number of different approaches were used by the participants which are explained in more detail below. The initially selected AQIs were presented to the audit committee and then reported on by the audit firm and management. CPAB had various touchpoints with the Pilot participants during 2016 to collect feedback and provide assistance, as required.
Figure 1 Summary of 2016 AQI Pilot Project CPAB call for Pilot participants
Pilot participants meet with CPAB and obtain background information
Pilot participants select AQIs and reporting mechanism/frequency
February 2016
March 2016
April - August 2016
Interim AQI reports provided to participants by audit firm, if applicable
CPAB holds Roundtable for AQI Pilot Participants
CPAB issues first report on AQI Pilot Project and releases call for additional participants
August - October 2016
November 2016
January 2017
Year-end AQI reports provided to participants by audit firm
CPAB’s Audit Quality Symposium
February - May 2017
May 2017
Where we are We are currently part way through the Pilot. All participants have selected their AQIs and most have received an interim on report on some indicators, as applicable. In November 2016, CPAB held a Roundtable with participants to obtain their initial observations. The Roundtable included audit committee chairs, management and lead audit partners from participating organizations. The discussion covered the following topics:
●
Benefits, challenges and surprises of using AQIs.
●
Usefulness of specific AQIs.
●
Integration of AQIs into regular audit committee processes.
●
Considerations for future use of AQIs and lessons learned.
2016 AQI Pilot Project
INTERIM REPORT
2
CPAB will continue to follow participant experiences in 2017 to complete a full annual cycle and get feedback on year-end AQI discussions. AQIs will also be on the agenda at CPAB’s Audit Quality Symposium (AQS) in May 2017. Several Pilot participants will attend the AQS to share their experiences with the broader director community.
PRELIMINARY OBSERVATIONS Process and practical considerations Participants began the AQI process with an open conversation among management, the audit firm and the audit committee chair. Participants considered the purpose of their participation in the Pilot and their main objective in using AQIs. A number of objectives were identified, including:
●
General oversight of the audit, including project management and monitoring of key audit risks.
●
External auditor evaluation in terms of both audit quality and client service.
●
Monitoring and managing the added value provided by the auditor.
The chosen objective(s) significantly influenced the selection of the specific AQIs. Participants noted that AQIs would not be the only tool available to the audit committees to meet their stated objective(s), but that instead, AQIs would be considered in conjunction with other sources of information. With the exception of the initial AQI conversation, reporting mechanisms and frequency varied. All participants agreed that the best time for the audit firm to present the initial list of AQIs to the audit committee was during the audit planning meeting, as part of the audit plan. All participants also had a separate AQI report which was produced by the auditor for the audit committee. After the initial presentation, reporting on AQIs varied by participant, and also by specific indicator. Frequency of reporting varied between quarterly, semi-annually, annually, or a combination of the three. At this stage of the Pilot, year-end reporting on AQIs has not yet occurred, however, participants have indicated that they expect year-end reporting to take place either at or after the year-end audit results meeting.
Involvement of management Management’s involvement in the AQI process is of importance to note. All participants included management in AQI discussions from the beginning and management’s involvement shaped which AQIs were selected. Some participants reported that management’s involvement resulted in one of the key benefits of AQIs - the ability to have an open and honest conversation about expectations for each stakeholder involved in the audit.
Specific Audit Quality Indicators Participants were encouraged to select a limited number of indicators (5-10) in the first year of the Pilot. While Participants were provided with existing resources about AQIs, no requirements were set regarding which AQIs should be selected. Instead, participants were encouraged to broadly consider how AQIs may be useful to them and to select their AQIs accordingly.
Number of AQIs selected by each participant Company
# of AQIs
Company A
9
Company B
10
Company C
9
In determining which AQIs to select, participants were encouraged to consider:
Company D
6
●
The unique nature of their business.
Company E
8
●
Specific areas of significant audit risk.
Company F
9
2016 AQI Pilot Project
INTERIM REPORT
3
●
Areas of the audit over which they would like more transparency or information.
●
Factors that are most meaningful to them when evaluating their external auditor.
●
Themes noted in the reports of external audit inspections including CPAB’s latest Public Report on Inspections.
●
AQIs that their external auditor is already tracking or considering for their own internal management purposes.
Ability of their external auditor to provide them with the information.
●
Participants reported that they also considered these additional factors:
●
Audit quality metrics their audit firm tracks for partner compensation purposes.
●
Information already provided by the audit firm in other reports (for example, as contractually required or requested for other purposes).
Output arising from the annual or comprehensive auditor evaluation tool, if used in the same year.
●
As a result, there was significant variety in the types of AQIs selected by the participants. This relates partially to differing objectives, but also to the innovative mindset of the participants, some of whom attempted to develop novel measures.
Example AQIs selected by Pilot participants Category
Example AQIs
Audit Execution
• • • • •
Firm Level Metrics
• Audit firm quality review results (internal and/or external) • Training and professional development • Independence
Management Indicators
• •
Engagement Team
• Staffing leverage • Experience of engagement team • Workload • Use of specialists • Engagement team turnover
Client Service Indicators
• Survey of audit committee members • Sharing of thought leadership / ideas / best practices with management and/or the audit committee
2016 AQI Pilot Project
INTERIM REPORT
Audit hours by phase Audit hours by risk Amount of audit work performed by centralized service centres Achievement of significant milestones in the audit Frequency and areas of challenge
Achievement of timing of agreed upon deliverables from management to the auditor Timely reporting of internal control deficiencies identified by management to the audit committee and/or external auditor
4
BENEFITS AND CHALLENGES Roundtable participants were asked to identify key benefits and challenges from the use of AQIs. Existing literature identified a number of benefits including more efficient and effective interactions between the audit committee and the auditor and assistance in auditor evaluation. Participants also identified these additional benefits as the most significant to them: 1. A better understanding of management, audit committee and external auditor expectations of their roles in the audit and responsibilities related to audit quality, particularly as a result of discussions about how to evaluate AQIs and what benchmarks should be set. 2. An improvement in the knowledge of, and engagement in, the audit process and audit quality by all members of the audit committee. These benefits enhance audit quality in two ways: 1. An enriched understanding of expectations among stakeholders improves coordination and cooperation in the execution of an audit. 2. A more informed and engaged audit committee results in improved oversight over the external audit process. Participants also noted other benefits such as an improvement in the project management of an audit through the discussion of key deliverables and milestones; an improved understanding of the factors that contribute to audit quality, and improved collaboration among the various stakeholders in the audit process. The most significant challenge identified by participants was the development of evaluation criteria. Other challenges identified included the fact that key contributing factors to audit quality, such as how an audit firm and engagement team are able to deal with the unexpected, are not easily quantifiable; reporting on AQIs requires a change in systems, processes and attitudes at the audit firm level which takes time; and that interpreting AQIs is a skill that audit committees will have to build.
KEY INSIGHTS Project management and audit quality The Roundtable highlighted the importance of project management to audit quality. A number of participants developed indicators related to tracking milestones or phases of the audit. These indicators typically held the audit firm accountable for meeting certain milestones by a specific date. For example, one audit firm provided the audit committee with a chart listing significant tasks (e.g. lead engagement partner sign-off on audit planning) and a date when the task was to be completed. Another audit firm provided dates when a certain amount of the audit work was to be completed (e.g. 55 per cent of the audit to be completed before year end). One audit firm noted the requirement to report on such milestones to the audit committee helped the engagement team stay focused on project management. Other participants noted that the discussion of such milestones highlighted the importance of management providing working papers to the audit firm on time.
2016 AQI Pilot Project
INTERIM REPORT
5
Management’s role in audit quality A number of participants developed AQIs to measure management’s role in audit quality. The inclusion of these indicators provides a broader perspective to the audit committee highlighting the contribution of management to the successful completion of a high quality audit. For example, two participants developed AQIs to measure the timeliness of deliverables due from management to the external auditor.
As a result of including a management AQI in their AQI profile, one audit committee discovered that over 40 per cent of management’s deliverables were provided an average of five days late to the auditor. This insight resulted in actions from management that positively impacted audit quality.
In 2017 CPAB will work with audit committees, management and the auditor to better understand measures and information that can help to highlight the important role that management plays in establishing the control environment and working together with the auditor to ensure a high quality audit.
Definitions and evaluation Many measures were selected by more than one participant, particularly the AQIs listed in the Engagement Team and Audit Execution categories. However, the way each measure was defined, reported and evaluated often differed, depending on the information capability of the audit firm and the needs of the specific audit committee. Participants identified the development of evaluative criteria as the most significant challenge to using AQIs. While the need for context was recognized by all, most participants agreed that additional qualitative information was not enough to make a definitive conclusion. Examples of evaluation criteria included comparison to prior year(s), a budget, an objective target, a benchmark, or to an industry standard. Most participants evaluated AQIs by comparing to budget or to an agreed upon target due to the lack of industry or objective standards and the fact that this was the first year the participants used AQIs. Participants agreed that historical trending would be useful in evaluating a current year AQI and that such trending would come from continued use of such AQIs in future years.
Diversity in perceived usefulness Roundtable participants were asked to rate each of the indicators on a five point scale (5 being extremely useful and 1 being not useful at all), regardless of whether they had selected the indicator themselves. Five indicators consistently achieved an average score higher than four: Audit hours by risk, Use of specialists, Audit hours by phase, Staffing leverage, and Experience of engagement team. Reactions to all other indicators were mixed from mildly useful to extremely useful. This illustrates the diverse perspectives of participants and their differing needs and goals.
2016 AQI Pilot Project
INTERIM REPORT
6
Workload AQI Example This in-depth look at the Workload Indicator provides an illustrative example of how Pilot participants had varying perspectives on specific indicators. This indicator was selected by three of the six Pilot participants. Example definitions
Example evaluative criteria
Absolute and percentage of hours dedicated to Company X, other client commitments and internal activities by Lead Audit Partner and Engagement Quality Control Reviewer
“The Lead Audit Partner and Engagement Quality Control Reviewer will devote X and X hours respectively to the client. Partners are normally expected to work X number of client service hours annually.”
Number of hours of the Lead Audit Partner, Engagement Quality Control Reviewer, Other Key Partners and Audit Managers
“Our audit firm will meet the budgeted figures as listed below, per staffing level and specialist group:” Chart provided lists the following figures per staffing level and specialist group: (1) Planned average hours to be spent on the client in the current year (2) Planned average total client service hours in the current year (3) Actual average total client service hours in the previous year
When discussing the usefulness of the workload indicator, participants had varying perspectives ranging from mildly valuable to very valuable. Some audit committee chairs felt that the workload of key partners was not important as long as the partners continued to deliver a quality audit to their specific engagement. Other audit committee chairs said it is important to understand the amount of pressure their key partners are under as it related to their ability to make critical judgements. During the discussion, it was also noted that certain unique situations may result in increased interest in this indicator in a given year, such as: if it is a first year audit, if a significant transaction occurred during the year, or if a new audit partner is being rolled onto the engagement.
WHAT’S NEXT? AQIs are a useful tool for audit committees to broaden and deepen the dialogue around audit quality. CPAB supports increased awareness, discussion and collaboration around AQIs to develop good practices. To help advance this effort, CPAB will continue the AQI Pilot Project in 2017 and we encourage Canadian reporting issuers to participate. If you are interested, please contact Lindsay Colley, Director, CPAB at
[email protected].
ADDITIONAL INFORMATION For more information about CPAB’s AQI Pilot Project, please contact Lindsay Colley, Director, CPAB at
[email protected].
This publication is not, and should not be construed as, legal, accounting, auditing or any other type of professional advice or service. Subject to CPAB’s Copyright, this publication may be shared in whole, without further permission from CPAB, provided no changes or modifications have been made and CPAB is identified as the source. © CANADIAN PUBLIC ACCOUNTABILITY BOARD, 2017. ALL RIGHTS RESERVED www.cpab-ccrc.ca / Email:
[email protected]
2016 AQI Pilot Project
INTERIM REPORT
7
