Cloud updates and other technology advances have put new pressure on most technology companies—especially Software as a Service (SaaS) companies—to streamline roll-outs of updates and new product offerings. To meet customer needs, and to get features and products released to market faster, SaaS companies are turning to Agile frameworks to implement an iterative software development process and better streamline their updates and development. An Agile framework allows for more real-time testing, development and a faster delivery of product. However, from the perspective of an auditor performing a System and Organization Controls (SOC) examination, the process of gleaning details to determine control design suitability and operating effectiveness within an Agile environment could be a bit more challenging to the typical SOC approach if there’s a lack of documentation.
Requests for SOC reports for organizations utilizing an Agile environment are likely to rise. Due to the nature of their services, SaaS companies are often required to demonstrate the effectiveness of their controls over their customer’s data. It is very important that both auditors and software companies are aware of how Agile environments affect the SOC reporting process. The following highlights for both sides what specifically could be challenging.
A Brief History of Agile Frameworks
The Agile framework is based on the Agile Manifesto. The Manifesto came into being in the early 2000s as a method to speed up the delivery time within the software development process, and to provide an efficient communication loop between developers and end users. Agile frameworks used by project management tools such as Scrum and Extreme Programming (XP) purposely eschew documentation in favor of a more collaborative, continually evolving approach to testing and rolling out new applications or updates. Software development in Agile environments is conducted in “sprints” or short-turn windows, usually about two weeks, that result in a new feature or product being released to the customer. Not to mention Agile has a defining feature known as Minimum Viable Product (MVP). MVP is a concept that stresses the impact of learning in new product development. MVP is defined as that version of a new product that allows a team to collect the maximum amount of validated learning about customers with the least amount of effort, an added benefit of using an Agile framework.
Pre-Agile Manifesto, SaaS companies followed a waterfall model of software development. Each stage was completed before the next began. The waterfall model also required extensive documentation, which made it easier for auditors to review, ascertain segregation of duties, and chronologically follow the process. Software developed under the waterfall model had a clear paper trail. Change orders were documented, and for each stage, sign offs are evident. Not so with the work and test and rework as you go approach of the Agile framework. Because of its heavy reliance on automated input and collaboration, there are risks that too little is documented to demonstrate effective processes to auditors.
Although this is the case, it doesn’t mean that organizations utilizing an Agile framework approach aren’t careful about the best practices of development and change management. The approach is just different and requires a careful process design to capture key information during development.
The Challenge for Auditors
Auditors look for internal controls over the development process, such as the segregation of duties to reduce the risk of fraud or the controls in place to protect sensitive information. They may need to do some additional inquiries, observations, and inspections to find the information they need if the SaaS company uses an Agile framework. In the collaborative environment, it is not always clear who had the final say in approving a new feature. Change orders may also not be clear because the Agile environment may not require formal, written change order requests. Further complicating matters, there could be turnover among the software personnel who worked on the project so that if an auditor has a question about a change order request, the person(s) who implemented it may no longer be with the company. The paper trail under the waterfall model is distinctly auditor-friendly in regards to documentation and management’s oversight in comparison to that of the Agile environment. With limited documentation, the Agile environment creates a degree of difficulty for the auditor to achieve the assurance needed to attest to the mitigation of risks and the effectiveness of controls when examining the organization.
How to Prepare for a SOC Examination if Development is in the Agile Environment
SOC examinations of companies utilizing Agile environments are only going to become more common. The industry is shifting toward a more nimble approach to software development, and auditors will need to adjust. At the same time, surges in cybersecurity threats and other information security issues have increased the demand for various types of SOC reports, including SOC reports for information security procedures under SOC 2.
If utilizing an Agile framework in development, SaaS companies and their auditors should be prepared for some of the following changes in the examination process.
Change Order Requests
Instead of written requests or change orders, auditors may need to request system logs, or change management tickets if a ticketing system is used to manage changes. Acceptance criteria can also help auditors understand the SaaS controls in place to ensure product updates. Definition of Done (DoD) records can be helpful as the contents are usually a checklist of features and activities. Examples include: writing code, coding comments, unit testing, integration testing, security testing, release notes, design documents, and testing of other Non-Functional Requirements (NFRs). These DoD records can assist an auditor in achieving reasonable assurance.
Formal Testing
Formal testing may not be happening at the same point in the project in an Agile environment compared to the waterfall model, but controls may be just as effective at protecting against fraud or information security vulnerabilities. Formal testing should have data that supports the conclusions made from the tests performed. It will be incumbent on the auditor to be creative to understand the service organization and its Agile environment to achieve the desired assurance. It will also be equally important for service organizations to be diligent in recording every step of their development process to create a measurable examination trail.
Stakeholders
In the past, the SaaS product owner may have been the auditor’s primary contact. The Agile environment may require a broader pool of stakeholders. Product development is collaborative, so the developers that had significant contributions to certain components of the project may need to be included in the SOC conversation.
Segregation of Duties
At the end of the day, the segregation of duties may not be the same in the waterfall environment as in the Agile environment, but the objective—to deliver a sound product—is still there. In Agile environments where one individual performs multiple roles (for example application development and implementing updates to production systems), duties should be assigned such that no one individual has end-to-end control of a process without an independent checkpoint. The Agile environment strives to maintain code integrity throughout the development project so that a single change can’t override or disrupt code that has previously been vetted and approved. Although this is the case, it will be advantageous to the SOC examination process for both the auditor and the service organization if the service organization establishes clear timelines and interjections of human input and oversight (e.g. a committee or a board), on top of the vast automated input provided. This would be ideal for every sprint as it mitigates against the issues of segregation of duties that are always present when utilizing an Agile environment. Automation being laid on top of the human-based work creates that other, much-needed layer of checks and balances.
Culture Change is Here to Stay
Auditors and SaaS companies should be in continuous communication during the SOC examination process, particularly for the first examinations conducted for organizations with Agile environments. In many cases, the underlying principles of quality control and assurance are still at play, but they are taking different forms or have slightly different names than they would in a waterfall environment. The outcome can be the same. It’s all in the approach and skill set of the auditor to achieve reasonable assurance, and the responsibility of the auditee to document and record to create an examination trail for the auditor.
For specific comments, questions, or concerns about SOC examinations, please contact us.
Copyright © 2019, CBIZ, Inc. All rights reserved. Contents of this publication may not be reproduced without the express written consent of CBIZ. This publication is distributed with the understanding that CBIZ is not rendering legal, accounting or other professional advice. The reader is advised to contact a tax professional prior to taking any action based upon this information. CBIZ assumes no liability whatsoever in connection with the use of this information and assumes no obligation to inform the reader of any changes in tax laws or other factors that could affect the information contained herein.
CBIZ MHM is the brand name for CBIZ MHM, LLC, a national professional services company providing tax, financial advisory and consulting services to individuals, tax-exempt organizations and a wide range of publicly-traded and privately-held companies. CBIZ MHM, LLC is a fully owned subsidiary of CBIZ, Inc. (NYSE: CBZ).