5 Operational Acceptance Testing OAT
What is Acceptance Testing?
Acceptance testing is the culmination of the software testing process. Duringacceptance testing, testers ensure that the software can handle real-life userrequirements according to the specifications detailed by the product owner.As the last phase of the testing process, following system testing, acceptancetesting determines whether a given software is acceptable for delivery or not.Acceptance testing can be carried out by an in-house team of users thatweren’t involved in the development of the software. This is also known asinternal acceptance testing or alpha testing, and it can involve the use ofuser stories that may include several acceptance tests to verify thecorrectness of the software’s functionality.Additionally, acceptance testing can directly involve the end-users, who helptesters assess accurately whether the product is ready to be launched. Thismethod, known as external acceptance testing, user acceptance testing or betatesting, provides valuable feedback about the system’s performance when in thehands of the end-user.Both methods share a similar number of steps that need to be carried out inorder to implement acceptance testing successfully.
Steps to perform Acceptance Testing:
1 – Planning: The first step is to outline a strategy for acceptance testingthat guides the team in interpreting the results and ensures that the wholeprocess occurs within the desired timeframe. Planning should ideally takeplace early on, when the requirements are written. A common problem is thatplanning is done too late, worst case being a couple of weeks beforeacceptance testing starts.2 – Designing test cases: Test cases are used to outline real-life functionalscenarios of the software being used by the end-user. They are written inclear and simple language to ensure that the scenarios present are unambiguousand make the process easier. Test cases should be written in parallel withrequirements, sometimes as acceptance criteria.3 – Choosing a team of testers: The organisation selects a team of testerseither from within the company itself or by recruiting a number of end-usersonline through beta testing. Testers should be wisely selected in order torepresent different parts of the business. To test economy or financialfeatures for example, specialists from accounting should be involved intesting.4 – Executing test cases and documenting results: The testing team carries outthe test cases prepared and any difficulties or bugs that crop up during theirexperience are logged in the testing documentation with any additionalcomments that explain the circumstances under which the system did not performas expected.5 – Fixing bugs: The development team makes the necessary changes to thesoftware code in order to eliminate bugs. When the bugs have been fixed, theacceptance testers verify that the bugs are corrected in a proper way. This iscalled re-testing. Each bug report is checked for correctness. When acorrected version is delivered to acceptance testing, also a certain amount ofregression testing is done. The purpose of regression testing is to make surethat no bugs have been introduced when developers corrected the bugs reported.This can be done using a set of regression test cases, typically importantflows or parts where bugs often occur based on previous experience.6 – Prepare for launch: When the software is certified to be as bug-free aspossible and that is fulfills user requirements as specified by the productowner, the testing team declares the software to be acceptable for launch andthe necessary preparations are made to roll out the finished product in themarket.Acceptance testing in all its forms is a crucial part of the software testingprocess since it indicates to the team whether the software’s functionalitycoincide with the requirements made at the beginning of the project.This level of testing proves whether the software is suited to the real-worldcircumstances under which it will be used by the end-users and helps to revealany bugs that need to be fixed before the product is shipped.Acceptance testing ultimately guarantees that the end-user will be satisfiedwith the final product and minimises the risk for the company behind thesoftware of having to make any fixes to the software after it is releasedpublicly.
Acceptance Test Status Report
Acceptance Test Report should always summarize the acceptance tests that areperformed along with their results. It should be addressed to all theidentified stakeholders who are a part of Acceptance Testing Phase. Once theexecution of Acceptance Tests have started, the progress should be reported ona day-to-day basis.
Acceptance Test Summary Report
This is the report which summarizes the status of the entire AcceptanceTesting phase. This involves details like testing activities conducted,references to criteria met, requirement specifications, business rules,execution results, planned schedules, deviations, etc.
Acceptance Testing in Agile
In Agile, Acceptance Criteria of each User story is targeted for AcceptanceTests, i.e., Acceptance tests are derived from the Acceptance criteria of auser story. Each Acceptance Criteria can have one or more Acceptance Tests tocover the scenario.Acceptance Tests are usually designed by a QA who is the Subject MatterExpertise in the area. Acceptance Testing in Agile starts much early whencompared to the other approaches, usually within the sprints itself.It is performed very frequently as each sprint will have new User storiescoming up and also enhancements/continuation of the previous stories.Acceptance Testing is performed at two different stages in Agile: * When the feature is created and in its initial stage – basic. * When the feature is integrated and stabilized with the other features of the product.Each user story here has to undergo Acceptance Test and should be passed forconsideration. Any failures in Acceptance test should be considered as a highpriority and fixed immediately, this, in turn, will have the acceptance testto execute it.Story points are given to each user story based on the success of Acceptancetest results for each of the acceptance criteria. Acceptance Testing alsodefines the completion at User Story level, stating that the Acceptancecriteria for the story are met.
Getting customer-focused acceptance criteria through team planning
As a software test team member, you should actively participate in allplanning and design meetings. One way to do so is to look at the existing userstories that are coming up and write a test prototype that includes acceptancecriteria. Come up with testing workflows and stories through admissible means.Use a spreadsheet, matrix or text to create test scenarios for all theworkflows that end users are expected to follow. Include an outline of whatthe test objectives are and what type of test is planned: unit, automated ormanual. Make it a simple and concise test plan, and in a format the team canquickly review.If a tester takes the initiative to pre-plan testing in a simple, easy-to-useformat, then that paints a picture for other team members of the test team’sunderstanding of how the function is going to work for end users. It alsobrings out any contradictions and misconceptions so the team can discuss them,and makes the tester an active participant in all phases of development ratherthan just the testing phase.Testers can influence code design as equally as they test it. The goal isn’tsimply to make code pass unit tests. Real testing takes more than that.Testers must participate in and influence the development design by creatingadditional conversations and adding test plans to the acceptance criteria foreach story.
What is Acceptance Testing?
Once the System Testing process is completed by the testing team and issigned-off, the entire Product/application is handed over to the customer/fewusers of customers/both, to test for its acceptability i.e.,Product/application should be flawless in meeting both the critical and majorBusiness requirements. Also, end-to-end business flows are verified similar asin real-time scenario.The production-like environment will be the testing environment for AcceptingTesting (Usually termed as Staging, Pre-Prod, Fail-Over, UAT environment).This is a black-box testing technique where only the functionality is verifiedto ensure that the product meets the specified acceptance criteria (no needfor design/implementation knowledge).
5) Operational Acceptance Testing (OAT)
This is to assess the operational readiness of the Product and is a non-functional testing. It mainly includes testing of recovery, compatibility,maintainability, technical support availability, reliability, fail-over,localization etc.OAT mainly assures the stability of the Product before releasing it to theproduction.
Who does Acceptance Testing?
For Alpha type, only the members of the organization (who developed theProduct) perform the testing. These members are not directly a part of theproject (Project managers/leads, developers, testers). Management, Sales,Support teams usually perform the testing and provide feedback accordingly.Apart from the Alpha type, all other acceptance types are generally performedby different stakeholders. Like customers, customer’s customers, specializedtesters from the organization (not always).It is also good to involve Business Analysts and Subject Matter Expertisewhile performing this testing based on its type.
Similar to Product test cases, we do have acceptance tests. Acceptance testsare derived from User stories’ acceptance criteria. These are usually thescenarios that are written at the high-level detailing on what the Product hasto do under different conditions.It does not give a clear picture on how to perform tests, as in test cases.Acceptance tests are written by Testers who have a complete grip on theProduct, usually Subject Matter Expertise. All the tests written are reviewedby a customer and/or business analysts.These tests executed during acceptance test. Along with acceptance tests, adetailed document on any set-ups to be done has to be prepared. It shouldinclude every minute details with proper screenshots, set-up values,conditions, etc.
Acceptance Test Bed
Test Bed for this testing is similar to a regular test bed but is a separateone. Platform with all the required hardware, software, operating products,network set-up & configurations, server set-up & configurations, database set-up & configurations, licenses, plug-ins, etc., have to be set up very muchalike the Production environment.Acceptance test bed is a platform/environment where the designed acceptancetests will be executed. Before handing over the Acceptance test environment tothe customer, it is a good practice to check for any environmental issues andstability of the Product.If there is no separate environment set up for acceptance testing, a regulartesting environment can be used for that purpose. But here, it will be messyas the test data from regular System Testing, and the real-time data fromacceptance testing are maintained in a single environment.Acceptance test bed is usually set up on the customer-side (i.e., in thelaboratory) and will have restricted access to the development and testingteams.Teams will be required to access this environment through VMs/or specificallydesigned URLs using special access credentials, and all the access to thiswill be tracked. Nothing on this environment has to be added/modified/deletedwithout the customer’s permission, and they should be notified of the changesthat are made.
Acceptance Testing Process
In V-Model, AT phase is in parallel to the Requirements phase.Actual AT process goes as shown below:Business Requirements AnalysisBusiness requirements are analyzed by referring all the available documentswithin the project.Some of which are: * System Requirement Specifications * Business Requirements Document * Use Cases * Workflow diagrams * Designed data matrixDesign Acceptance Test PlanThere are certain items to be documented in the Acceptance Test Plan.Let’s take a look at some of them: * Acceptance Testing strategy and approach. * Entry and exit criteria should be well-defined. * The scope of AT should be well-mentioned and it has to cover only the business requirements. * Acceptance test design approach should be detailed so as anyone writing tests can easily understand the way in which it has to be written. * Test Bed set up, actual testing schedule/timelines should be mentioned. * As testing is conducted by different stakeholders, details on logging bug should be mentioned as the stakeholders may not be aware of the procedure followed.Design and Review Acceptance TestsAcceptance tests should be written at a scenario level mentioning what has tobe done (not in-detail to include how to do). These should be written only forthe identified areas of scope for business requirements, and each and everytest has to be mapped to its referencing requirement.All the written acceptance tests have to be reviewed to achieve high coverageon business requirements.This is to make sure that any other tests apart from scope mentioned are notinvolved so that testing lies within the scheduled timelines.Acceptance Test Bed Set upTest Bed should be set up similar to a Production environment. Very high-levelchecks are required to confirm on environment stability and usage. Share thecredentials to use the environment only with a stakeholder who is performingthis testing.Acceptance Test Data Set UpProduction data has to be prepared/populated as test data in the systems.Also, there should be a detailed document in such a way that the data has tobe used for testing.Do not have the test data like TestName1, TestCity1, etc., Instead haveAlbert, Mexico, etc. This gives a rich experience of real-time data andtesting will be up-to-the-point.Acceptance Test ExecutionDesigned Acceptance tests have to be executed on the environment at this step.Ideally, all the tests should pass at the first attempt itself. There shouldbe no functional bugs arising out of Acceptance testing, if any, then theyshould be reported at a high priority to be fixed.Again, bugs fixed have to be verified and closed as a high priority task. Testexecution report has to be shared on a daily basis.Bugs logged in this phase should be discussed in a bug-triage meeting and hasto undergo Root Cause Analysis procedure. This is the only point whereacceptance testing assess whether all the business requirements are actuallymet by the product or not.Business DecisionThere comes out a Go/No-Go decision for the product to be launched inProduction. Go decision will take the product ahead to be released to themarket. No-Go decision marks the product as Failure.Few factors of No-Go Decision: * Poor Quality of the product. * Too Many open Functional Bugs. * Deviation from business requirements. * Not up to the market standards and needs enhancements to match the current market standards.