ApproachGlossary.csv

Name,Approach Category,Definition,Parent(s),Synonym(s),Notes
0-Switch Testing,"Technique (inferred from state transition testing and IEEE, 2021c, p. 19)","Testing that ""cover[s] valid single transitions in the state model"" (IEEE, 2021c, p. 19)","State Transition Testing (IEEE, 2021c, p. 19)","Single Transition Testing (IEEE, 2021c, p. 19)",
1-Switch Testing,"Technique (inferred from state transition testing and IEEE, 2021c, pp. 19-20)","Testing ""requires pairs of transitions to be exercised"" (IEEE, 2021c, p. 20)","N-Switch Testing (IEEE, 2021c, p. 20)",,
A/B Testing,"Practice (IEEE, 2022, Fig. 2), Type (inferred from usability testing)","Testing ""that allows testers to determine which of two systems or components performs better"" (IEEE, 2022, pp. 1, 36)","Statistical Testing (IEEE, 2022, pp. 1, 36), Usability Testing (Firesmith, 2015, p. 58)","Split-Run Testing (IEEE, 2022, pp. 1, 36)","""Can be time-consuming, although tools can be used to support it"", ""is a means of solving the test oracle problem by using the existing system as a partial oracle"", and is ""not a test case generation technique as test inputs are not generated"" (IEEE, 2022, p. 36)"
"Absolute Correctness Testing (inferred from absolute correctness (Lahiri et al., 2013, p. 345) and correctness testing)",Type (inferred from correctness testing),,Correctness Testing,,
Acceptance Testing,"Level (IEEE, 2022, pp. 12, 21-22, 26-27; 2021a, pp. 6, 11; 2021b, p. 41; 2021c, p. 6; 2017, p. 467; Washizaki, 2024, p. 5-7; Hamburg and Mogyorodi, 2024; Black, 2009, p. 18; Peters and Pedrycz, 2000, Tab. 12.3; van Vliet, 2000, p. 439; Hetzel, 1988, pp. 9-12, 32, 73, 122, 139-143, 149; Sakamoto et al., 2013, p. 344)","""Testing conducted to enable a user, customer, or other authorized entity to determine whether to accept a system or component"" and whether it ""satisfies its acceptance criteria"" and/or ""the contractual requirements are met"" (IEEE, 2017, p. 5; similar in 2022, p. 13; 2021a, p. 6; 2021c, p. 6; Sakamoto et al., 2013, p. 344) ""with respect to user needs, requirements, and business processes"" (Kam, 2008, p. 42; similar in Peters and Pedrycz, 2000, Fig. 12.41)","Dynamic Testing (IEEE, 2021a, pp. 11, 29), Requirements-based Testing (Peters and Pedrycz, 2000, Tab. 12.3), Formal Testing (Hetzel, 1988, pp. 15, 150; Kam, 2008, p. 42), V-Model Testing (Gerrard, 2000a, p. 9), W-Model Testing (Figs. 3-5), User Testing (can be) (p. 42), Independent Tester Testing (can be) (p. 52), Usability Testing (often) (van Vliet, 2000, p. 439), System Testing (implied by p. 439), Integration Testing (implied by Sakamoto et al., 2013, p. 344)","Qualification Testing (Bourque and Fairley, 2014, p. 4-6), AT (Firesmith, 2015, p. 30), Red-Box Testing (incorrectly) (Sneed and Göschl, 2000, p. 3)","Related to validation testing (IEEE, 2017, p. 5); note that they make a distinction between ""acceptance testing"" and an ""acceptance test"". ""Emphasis is [not] on … compliance of the code against some specification (van Vliet, 2000, p. 439), although this may be indirectly used (Peters and Pedrycz, 2000, Tab. 12.3). May be done in a pre-production environment (IEEE, 2021b, p. 42)"
"Access Control Testing (Firesmith, 2015, p. 57; implied by Perry, 2006, pp. 40-41)","Type (implied by Firesmith, 2015, p. 57)",,"Security Testing (Firesmith, 2015, p. 57; Bas, 2024, p. 28)",,
Accessibility Testing,"Type (IEEE, 2022, pp. 1, 22; 2021b, p. 41; 2021c, p. 36, Tab. A.1; implied by its quality (ISO/IEC, 2011); Firesmith, 2015, p. 58)","Testing ""used to measure the degree to which a test item can be operated by users with the widest possible range of characteristics and abilities"" (IEEE, 2022, p. 1; 2013, p. 2), including those with disabilities (Kam, 2008, p. 42; OG Gerrard) ""to achieve a specified goal in a specified context of use"" (ISO/IEC, 2011)","Usability Testing (IEEE, 2022, p. 1; 2021c, Tab. A.1; 2017, p. 6; ISO/IEC, 2011; Firesmith, 2015, p. 58), Model-based Testing (IEEE, 2021c, pp. 36-37), Requirements-based Testing, Conformance Testing (p. 37), Regression Testing (can be) (Washizaki, 2024, p. 5-8)","User Assistance Testing? (ISO/IEC, 2023a; see ISO/IEC 25019)","Good to use when the software will be widely available or when it will be used by people with disabilities (IEEE, 2022, p. 45) and may be done in a pre-production environment (2021b, p. 42)"
"Acquisition Organization Testing (Firesmith, 2015, p. 37)",Practice?,,"Organization-based Testing (Firesmith, 2015, p. 37)",,
Ad Hoc Reviews,Approach,"Reviews ""performed informally without a structured process"" (Hamburg and Mogyorodi, 2024)",Reviews,,See ISO 20246
Ad Hoc Testing,"Practice (IEEE, 2013, p. 33), Technique (Washizaki, 2025, p. 5-14)","Testing where ""test cases are derived by relying on the software engineer's skill, intuition, and experience with similar programs"" to ""identify[] test cases that are not easily generated by more formalized techniques"" (Washizaki, 2025, p. 5-14) ""performed without test analysis [or] test design"" (Hamburg and Mogyorodi, 2024)","Experience-based Testing (Washizaki, 2025, p. 5-14), Informal Testing (Hamburg and Mogyorodi, 2024)",,
Adaptive Random Testing,"Technique (Washizaki, 2024, p. 5-12)","Random testing ""in which other input selection criteria direct the random input sampling"" (Washizaki, 2024, p. 5-12)","Random Testing (Washizaki, 2024, p. 5-12)",,Kind of a vague definition; what are these criteria?
Adversarial Testing,"Technique (Hamburg and Mogyorodi, 2024)","Testing ""based on the attempted creation and execution of adversarial examples [(""input[s] to an ML model … that result[] in the model outputting an incorrect result with high confidence"")] to identify defects in an ML model"" (Hamburg and Mogyorodi, 2024)",ML Model Testing,,See ISO 29119-11
Agent-based Testing,Approach,,"Web Application Testing, Object-Oriented Testing, Data Flow Testing, Functional Testing, Specification-based Testing (usually) (Kam, 2008, Tab. 1)",,
"Agile Testing (Firesmith, 2015, p. 29)",Practice?,"Testing performed in the context of the agile development model ""based on iterative development, frequent inspection and adaptation, and incremental deliveries"" (IEEE, 2017, p. 15; OG ISO/IEC, 2016)","Continuous Testing (Firesmith, 2015, p. 29; implied by Washizaki, 2024, p. 9-6), Test-driven Development (Washizaki, 2024, p. 11-10; Sangwan and LaPlante, 2006, p. 25), Lifecycle-based Testing (Washizaki, 2024, p. 10-5; OG [2, 3, 10]), Incremental Testing (Sangwan and LaPlante, 2006, p. 26), DevOps Testing? (implied by Washizaki, 2024, p. 10-7; OG [11]), At-the-Beginning Testing? (implied by Kam, 2008, p. 42)",,"Test cases may change to match changes to requirements specifications (implied by Washizaki, 2024, p. 10-5; OG [9, 10])"
"Algebraic Testing (Peters and Pedrycz, 2000, Fig. 12.2)",Approach,,"Specification-based Testing (Peters and Pedrycz, 2000, Fig. 12.2), Mathematical-based Testing",,
All Combinations Testing,"Technique (IEEE, 2022, p. 22; 2021c, pp. 2, 16, Fig. 2; Washizaki, 2024, p. 5-11)","Testing that covers ""all unique combinations of P-V pairs"" (IEEE, 2021c, p. 16)","Combinatorial Testing (IEEE, 2022, p. 22; 2021c, pp. 2, 16, Fig. 2; Washizaki, 2024, p. 5-11)",,"""The minimum number of test cases required to achieve 100% … coverage corresponds to the product of the number of P-V pairs for each test item parameter"" (IEEE, 2021c, p. 16). See Grindal et al., 2005"
All Rules Testing,Technique (inferred from coverage-based testing),"Testing that covers ""a set of formal business rules"" (Doğan et al., 2014, Tab. 13; OG Thummalapenta et al., 2013)","Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13), Formal Testing, Control Flow Testing?",,
All Transitions Testing,"Technique (inferred from state transition testing and IEEE, 2021c, p. 20)","Testing that ""cover[s] both valid transitions in the state model and 'invalid' transitions (transitions from states initiated by events in the state model for which no valid transition is specified)"" (IEEE, 2021c, p. 19)","State Transition Testing (IEEE, 2021c, p. 19; Kam, 2008, p. 15), Model-based Testing (IEEE, 2021c, p. 19)",,
All-C-Uses Testing,"Technique (IEEE, 2022, p. 22; 2021c, Fig. 2; Washizaki, 2025, p. 5-13)","Testing that exercises ""control flow sub-paths from each variable definition to each c-use of that definition (with no intervening definitions)"" (IEEE, 2021c, p. 27; similar in van Vliet, 2000, p. 425; Peters and Pedrycz, 2000, p. 479)","Data Flow Testing (IEEE, 2022, p. 22; 2021c, pp. 3, 27, Fig. 2; Washizaki, 2025, p. 5-13), All-Uses Testing (IEEE, 2021c, p. 28; Peters and Pedrycz, 2000, pp. 478-479; implied by van Vliet, 2000, p. 425), Control Flow Testing (IEEE, 2021c, pp. 27-28), All-DU-Paths Testing (p. 29; implied by Reid, 1996, Fig. 2), Structural Testing (implied by p. 6), All-C-Uses/Some-P-Uses Testing (implied by van Vliet, 2000, p. 433)","Sometimes spelled in all lowercase (IEEE, 2021c, pp. 3, 27, Fig. 2), All Computation Data Uses (implied by IEEE, 2017, p. 83)",
All-C-Uses/Some-P-Uses Testing,Technique (inferred from data flow testing),"All-c-uses testing where, if a definition is only used in predicates, at least one definition-clear path to a p-use must be executed (van Vliet, 2000, p. 425)","Data Flow Testing (van Vliet, 2000, p. 425), All-Uses Testing (implied by van Vliet, 2000, pp. 425, 433)",,
All-Definitions Testing,"Technique (IEEE, 2022, p. 22; 2021c, Fig. 2; Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49)","Testing that exercises ""control flow sub-paths from each variable definition to some use (either p-use or c-use) of that definition (with no intervening definitions)"" (IEEE, 2021c, p. 27; similar in van Vliet, 2000, p. 425)","Data Flow Testing (IEEE, 2022, p. 22; 2021c, pp. 3, 27, Fig. 2; Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49; van Vliet, 2000, p. 425), Control Flow Testing, Automated Testing (may be required) (IEEE, 2021c, p. 27), All-DU-Paths Testing (implied by Washizaki, 2025, p. 5-13), All-Uses Testing (implied by van Vliet, 2000, pp. 425, 433), All-C-Uses/Some-P-Uses Testing, All-P-Uses/Some-C-Uses Testing (implied by p. 433)","Sometimes spelled without a hyphen (Firesmith, 2015, p. 49) or in all lowercase (IEEE, 2021c, pp. 3, 27, Fig. 2), All-Defs Testing (van Vliet, 2000, p. 425; implied (without hyphen) by Doğan et al., 2014, Tab. 11)",
All-DU-Paths Testing,"Technique (IEEE, 2022, p. 22; 2021c, Fig. 2; Washizaki, 2025, p. 5-13)","Testing that exercises all ""loop-free control flow sub-paths from each variable definition to every use (both p-use and c-use) of that definition (with no intervening definitions)"" (IEEE, 2021c, p. 29; similar in 2017, p. 125; Washizaki, 2025, p. 5-13; Peters and Pedrycz, 2000, p. 479); paths containing simple cycles may also be required (van Vliet, 2000, p. 425)","Data Flow Testing (IEEE, 2022, p. 22; 2021c, pp. 3, 27, Fig. 2; Washizaki, 2025, p. 5-13; van Vliet, 2000, p. 425), Control Flow Testing (IEEE, 2021c, p. 29), Path Testing (implied by van Vliet, 2000, Fig. 13.17; Reid, 1996, Fig. 2), Structural Testing (implied by p. 6)","Sometimes spelled in all lowercase (IEEE, 2021c, pp. 3, 27, 29, Fig. 2), All Def-Use Testing (implied by Doğan et al., 2014, Tab. 11)","""The strongest [feasible] data flow testing criterion"" (Washizaki, 2025, p. 5-13; similar in van Vliet, 2000, pp. 432-433; Reid, 1996, p. 3, Fig. 2)"
"All-Input-GUI Testing (Doğan et al., 2014, Tab. 13; OG Mansour and Houri, 2006)",Technique (inferred from input-parameter testing),,"Web Application Testing, GUI Testing, Input-Parameter Testing?",,
All-P-Uses Testing,"Technique (IEEE, 2022, p. 22; 2021c, Fig. 2; Washizaki, 2025, p. 5-13)","Testing that exercises ""control flow sub-paths from each variable definition to each p-use of that definition (with no intervening definitions)"" (IEEE, 2021c, p. 28; similar in 2017, p. 332; van Vliet, 2000, p. 425; Peters and Pedrycz, 2000, p. 479)","Data Flow Testing (IEEE, 2022, p. 22; 2021c, pp. 3, 27, Fig. 2; Washizaki, 2025, p. 5-13; van Vliet, 2000, p. 425), All-Uses Testing (IEEE, 2021c, p. 28; Peters and Pedrycz, 2000, pp. 478-479; implied by van Vliet, 2000, pp. 425, 433), Control Flow Testing (IEEE, 2021c, p. 28), All-DU-Paths Testing (p. 29; implied by Reid, 1996, Fig. 2), Structural Testing (implied by p. 6), All-P-Uses/Some-C-Uses Testing (implied by van Vliet, 2000, pp. 425, 433)","Sometimes spelled in all lowercase (IEEE, 2021c, pp. 3, 27, 28, Fig. 2)",
All-P-Uses/Some-C-Uses Testing,Technique (inferred from data flow testing),"All-p-uses testing where, if a definition is only used in computations, at least one definition-clear path to a c-use must be executed (van Vliet, 2000, p. 425)","Data Flow Testing (van Vliet, 2000, p. 425), All-Uses Testing (implied by van Vliet, 2000, pp. 425, 433)",,
"All-Round-Trip-Paths Testing (inferred from Kam, 2008, p. 15)",Technique (inferred from state transition testing),,"State Transition Testing (Kam, 2008, p. 15)",,
"All-Transition-k-Tuples Testing (inferred from Kam, 2008, p. 15)",Technique (inferred from state transition testing),,"State Transition Testing (Kam, 2008, p. 15)",,
All-URL Testing,Technique (inferred from coverage-based testing),"Testing that aims to ""cover[] each URL of the application at least once"" (Doğan et al., 2014, Tab. 13; OG Sprenkle et al., 2005)","Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13), Control Flow Testing?",,"Can be decomposed into covering single URLs, URL seq2, URL names, URL seq2 names, URL names values, and URL seq2 names values (Doğan et al., 2014, Tab. 13; OG Sampath et al., 2005)"
All-Uses Testing,"Technique (IEEE, 2022, p. 22; 2021c, Fig. 2; Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49)","Testing that exercises ""control flow sub-paths from each variable definition to every use (both p-use and c-use) of that definition (with no intervening definitions)"" (IEEE, 2021c, p. 28; similar in 2017, p. 120; Peters and Pedrycz, 2000, pp. 478-479; van Vliet, 2000, pp. 424-425) and potentially to ""each of these uses' successors"" (van Vliet, 2000, pp. 424-425) so that each use is tested at least once (IEEE, 2021c, p. 29) ","Data Flow Testing (IEEE, 2022, p. 22; 2021c, pp. 3, 27, Fig. 2; Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49; van Vliet, 2000, p. 425), Control Flow Testing (IEEE, 2021c, p. 28), All-DU-Paths Testing (implied by Washizaki, 2025, p. 5-13; (only if there are infeasible paths?) van Vliet, 2000, pp. 432-433)","Sometimes spelled without a hyphen (Firesmith, 2015, p. 49; Doğan et al., 2014, Tab. 11) or in all lowercase (IEEE, 2021c, pp. 3, 27, 28, Fig. 2)",
Alpha Testing,"Level (IEEE, 2022, p. 22; inferred from acceptance testing), Type (implied by Firesmith, 2015, p. 58)","""Simulated or actual operational testing"" (Kam, 2008, p. 42) that is ""first stage of testing before a product is considered ready for commercial or operational use"" (IEEE, 2017, p. 17)","Acceptance Testing (IEEE, 2022, p. 22; Hamburg and Mogyorodi, 2024; Dennis et al., 2012, p. 455), Unscripted Testing (Washizaki, 2024, p. 5-8), Tester Testing (Firesmith, 2015, p. 39), Security Testing (Firesmith, 2015, p. 57), Operational Testing (Kam, 2008, p. 42), User as Tester Testing? (implied by Washizaki, 2024, p. 5-8)","Alpha Tester Testing? (Firesmith, 2015, p. 39)","""Often performed only by users within the organization developing the software"" (IEEE, 2017, p. 17), by ""a small, selected group of potential users"" (Washizaki, 2024, p. 5-8), or ""in the developer's test environment by roles outside the development organization"" (Hamburg and Mogyorodi, 2024). Related to beta testing (IEEE, 2017, p. 17; Washizaki, 2024, p. 5-8). ""Often employed for off-the-shelf software as a form of internal acceptance testing"" (Kam, 2008, p. 42)"
"Anomaly Testing (Peters and Pedrycz, 2000, Tab. 12.1)",Approach,,"Structure-based Testing, Static Testing (Peters and Pedrycz, 2000, Tab. 12.1)",,
"Anti-Spoofing Testing (Firesmith, 2015, p. 57)","Type (implied by Firesmith, 2015, p. 57)",,"Security Testing (Firesmith, 2015, p. 57)",,
"Anti-Tamper Testing (Firesmith, 2015, p. 57)","Type (implied by Firesmith, 2015, p. 57)",,"Security Testing (Firesmith, 2015, p. 57)",,
API Testing,Approach,"A specific form of interface testing that ""simulate[s] the use of APIs by end-user applications"", including ""generating parameters"", ""defining internal data"" (Washizaki, 2024, p. 5-10), and ""submitting requests"" (Hamburg and Mogyorodi, 2024)","Interface Testing (Washizaki, 2024, p. 5-10)","Application Program Interface Testing (Washizaki, 2024, p. 5-10)",
Application System Testing,Approach,"Testing that ""aims to cover complete testing of the [e-business] application with business oriented scenarios to ensure all features of the system meet their requirements"" (Gerrard, 2000b, p. 17)","Functional Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1, p. 13), Functionality Testing (2000a, Tab. 2; 2000b, Tab. 1, p. 17), System Testing, Dynamic Testing (2000a, Tab. 2; 2000b, Tab. 1), Scenario Testing (implied by 2000b, p. 17)","AST (Gerrard, 2000b, p. 17), Application Testing? (Firesmith, 2015, p. 22)","""Simply System Testing for E-Business systems"", although ""the navigation buttons … are not under the control of the application"" and ""the Web is 'stateless'"" (which is why things like cookies are used) (Gerrard, 2000b, p. 17)"
"Architect Testing (Firesmith, 2015, p. 39)",Practice?,,"Developer Testing (Firesmith, 2015, p. 39), Architecture-driven Testing",,
Architecture-driven Testing,Approach,"Testing that ""verifies"" the ""system conforms to [the specified] architecture"" (Firesmith, 2015, p. 33)",,,
"Assertion Checking (Firesmith, 2015, p. 31)",Practice?,"Testing specifications to ensure that they are not violated (Chalin et al., 2006, p. 343)","Model-based Testing, Fault-based Testing (Mesbah and van Deursen, 2009, p. 213), Ongoing Built-In Testing (can be), Self-Testing (can be) (Firesmith, 2015, p. 31), DOM Testing (can be), Accessibility Testing (can be), Link Discoverability (Testing) (can be), Security Testing (can be) (Mesbah and van Deursen, 2009, p. 213), State Testing (can be), State Transition Testing (can be) (p. 214), Formal Modular Verification? (implied by Chalin et al., 2006, p. 342), Specification-based Testing?","Invariant-based Testing (implied by Doğan et al., 2014, pp. 184-185; Mesbah and van Deursen, 2009)","""Types [of assertion] include input assertion, loop assertion, output assertion"" (IEEE, 2017, p. 30), which may imply subapproaches. See Roest et al., 2010"
"Asynchronous Testing (Jard et al., 1999)",Practice?,"Testing in which the test environment, such as ""a network connection"", comes ""between the tester and the IUT"" (Jard et al., 1999, p. 26)","Test Environment Testing (implied by Jard et al., 1999, p. 26)",,"""Error-prone"" (Jard et al., 1999, p. 25)"
Attacks,"Practice (IEEE, 2022, p. 34; 2013, p. 33), Technique (implied by Hamburg and Mogyorodi, 2024)","""Action[s] or interaction[s] with the system or its environment"" that produce ""a fault or an error, and thereby possibly … a failure, or an adverse consequence"" (IEEE, 2019a, p. 7) by ""exploiting a specific … way of thinking about how and why software fails … [which] can be behaviour-based"" (2022, p. 34)","Experience-based Testing (IEEE, 2022, pp. 4, 34; 2021c, p. 4), Behavioural Testing (implied by 2022, p. 34), Privacy Testing (implied by Washizaki, 2024, p. 5-10), Disaster/Recovery Testing (implied by Bas, 2024, p. 26), Specification-based Testing (can be) (IEEE, 2022, p. 34; implied by Patton, 2006, pp. 87-89)","Fault Attacks (implied by Hamburg and Mogyorodi, 2024)","""Specific failures"" may be targeted (Hamburg and Mogyorodi, 2024). Attacks can be ""malicious"" (IEEE, 2019a, p. 7) when they are ""attempts to gain unauthorized access to system services, resources, or information or … compromise system integrity, availability, or confidentiality"" (NIST, 2023, p. 160)"
"At-the-Beginning Testing (inferred from at-the-end testing (Firesmith, 2015, p. 29) and Kam, 2008, p. 41)",Practice?,,"Static Testing?, W-Model Testing? (Gerrard, 2000a, Fig. 4), Incremental Testing?, Continuous Testing?","Early Test Case Preparation? (Gerrard, 2000a, Fig. 4)","Not described by any source; implied by at-the-end testing (Firesmith, 2015, p. 29) and early test case preparation (Gerrard, 2000a, Fig. 4)"
"At-the-End Testing (Firesmith, 2015, p. 29)",Practice?,"Testing that occurs as ""the last phase of the development life cycle"" (Gerrard, 2000a, p. 8)","Waterfall Testing (Firesmith, 2015, p. 29)",,"""The most costly and least effective way of performing testing"" (Gerrard, 2000a, p. 8)"
Audio Testing,Approach,"""Testing to determine if the game music and sound effects will engage the user in the game and enhance the game play [sic]"" (Hamburg and Mogyorodi, 2024)",,,
Audits,Practice?,"""Independent examination[s] of … work products to assess compliance with specifications, standards, contractual agreements, or other criteria"" (IEEE, 2017, p. 36; OG ISO/IEC, 2016; 2015, p. 4; OG ISO, 2010) ""often mandated to be performed by third parties"" (Washizaki, 2024, p. 12-14; similar in Hamburg and Mogyorodi, 2024)","Static Analysis? (Washizaki, 2024, p. 12-13), Static Testing, Compliance Testing (implied by IEEE, 2022, p. 44), Independent Test Organization Testing?",,"Also described as testing ""used to verify compliance with standards"" (IEEE, 2022, p. 44)"
Automated Testing,"Practice (IEEE, 2022, pp. 20, 22)","Testing ""executed by … test automation tool[s]"" (IEEE, 2022, p. 35), ""robots, … [or] other test execution engines"" (IEEE, 2022, p. 6; 2016, p. 3) such as test drivers (Gerrard, 2000a, p. 11)","Scripted Testing (IEEE, 2022, p. 33), Developer Testing (can be) (Gerrard, 2000a, p. 11)","Automatic Testing? (implied by Doğan et al., 2014, pp. 184-185, Tab. 21)","Often cost-effective ""if a set of test cases is going to be executed 5 or more times"" (IEEE, 2022, p. 35) but ""must be automated as much as possible throughout the entire software delivery process, including throughout development and operations"", and at all levels (Washizaki, 2024, p. 6-13); it is time-consuming to ""automate manual tests later"" (Gerrard, 2000a, p. 11). It is also ""typically highly technical and tool dependent""; ""automated test scripts may either be generated automatically by a framework or developed manually by a test automation specialist"" (IEEE, 2016, p. 7). See ""automated verification system"" (IEEE, 2017, p. 37)"
Availability Testing,"Type (implied by its quality (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1; 2017, pp. 38, 361; OG ISO/IEC, 2015; 2015, p. 8; OG NATO AEP-67))","Testing the ""capability of a product to be operational and accessible when required for use"" (ISO/IEC, 2023a) or ""to perform its required function at an agreed instant or over an agreed period of time"" (IEEE, 2017, p. 38; OG ISO/IEC, 2015), usually represented as a ratio, percentage (IEEE, 2017, p. 38), or proportion (ISO/IEC, 2023a) representing uptime (Gerrard, 2000b, p. 26)","Reliability Testing (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1; Washizaki, 2024, p. 7-10), Security Testing (IEEE, 2017, pp. 404-405; 2015, p. 8; OG NATO AEP-67; Washizaki, 2024, p. 5-9), Backup Testing (Bas, 2024, p. 14), Disaster/Recovery Testing (p. 26), Non-functional Testing (Bertolino et al., 2019, p. 93:11, Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Post-Deployment Monitoring (2000a, Tab. 2; 2000b, Tab. 1, p. 32), Automated Testing, Dynamic Testing (2000a, Tab. 2; 2000b, Tab. 1), Dependability Testing (if it exists) (IEEE, 2024, pp. 14, 21, 23, 48-59; 2017, p. 375; ISO/IEC, 2023a)",,"Can be supported by ""failover or duplication of systems"" (ISO/IEC, 2023a), ""diverse routing"", multiple servers, middleware, or ""distributed object technology that handles load balancing and rerouting of traffic in failure scenarios"" (Gerrard, 2000b, p. 26). Can be measured as a percentage of uptime (IEEE, 2024, p. 14). Related to robustness testing, error tolerance testing, and fault tolerance testing (IEEE, 2017, p. 38)"
"Axiomatic Testing (Peters and Pedrycz, 2000, Fig. 12.2)",Approach,,"Specification-based Testing (Peters and Pedrycz, 2000, Fig. 12.2), Formal Testing",,
Back-to-Back Testing,"Practice (IEEE, 2022, p. 22)","Testing ""whereby an alternative version of the system is used to generate expected results for comparison from the same test inputs"" (IEEE, 2022, p. 2) or where ""two or more variants of a program are executed with the same inputs, the outputs are compared, and errors are analyzed in case of discrepancies"" (2010, p. 30; similar in Hamburg and Mogyorodi, 2024; OG Spillner)","Non-functional Testing (Washizaki, 2024, p. 5-9)","Differential Testing (IEEE, 2022, p. 2)","""Not a test case generation technique as test inputs are not generated"" (IEEE, 2022, p. 35). Similar to testing using a pseudo-oracle (Barr et al., 2015, p. 515) and related to mutation testing (IEEE, 2010, p. 30)"
Backup and Recovery Testing,"Type (IEEE, 2013, p. 2)","Testing ""that measures the degree to which [the] system state [at ""a certain point in time"" (Bas, 2024, p. 27)] can be restored from backup within specified parameters of time, cost, completeness, … accuracy[, backup age, and downtime (p. 25)] in the event of failure"" (IEEE, 2013, p. 2)","Reliability Testing (IEEE, 2013, p. 2), Continuity Testing (implied by Washizaki, 2025, p. 6-8; Koomen et al., 2006, p. 50), Security Testing (can be) (Bas, 2024, pp. 25-26), Compliance Testing (can be) (p. 25), Performance Testing (implied by pp. 25, 27)","Backup and Restore Testing? (implied by Washizaki, 2025, p. 6-8; Koomen et al., 2006, p. 50)","Nonatomic; captures both ""backup"" and ""recovery"". Seems to be what is meant by ""recovery testing"" in the context of performance"
Backup Testing,Approach,"Testing that ""a copy of data[ can be] stored in another place … and used for recovery if the original data is lost or … corrupted … [or] for retrieving the earlier versions of data if needed"" (Bas, 2024, p. 15)","Continuity Testing (Washizaki, 2025, p. 6-8; Koomen et al., 2006, p. 50), Requirements-based Testing (Bas, 2024, pp. 14, 23), Backup and Recovery Testing, Distributed Testing (can be) (p. 15)",,"""Should be constantly rehearsed as changes to the production environment are made"" (Washizaki, 2024, p. 6-8). Could potentially have subapproaches based on different kinds of backups which can be combined: full, incremental, differential, passive (mirroring), and/or distributed (clustering) (Bas, 2024, pp. 15-16), and can be local (pp. 16, 23), offsite (p. 16), or cloud-based (pp. 16-17, 23)"
Backup/Recovery Testing,"Type (IEEE, 2021c, p. 37)","Testing that determines the ability ""to restor[e] from back-up memory in the event of failure, without transfer[ing] to a different operating site or back-up system"" (IEEE, 2021c, p. 37)","Disaster/Recovery Testing (IEEE, 2021c, p. 37; implied by Bas, 2024, p. 25)","Backup and Recovery Testing? (IEEE, 2013, p. 2)",
"Backwards Compatibility Testing (Firesmith, 2015, p. 53)","Type (implied by Firesmith, 2015, p. 53)",,"Compatibility Testing (Firesmith, 2015, p. 53)",,
Base Choice Testing,"Technique (IEEE, 2022, p. 22; 2021c, pp. 2, 17, Fig. 2)","Testing ""based on … representative or typical value[s] for … [given input] parameter[s]"" (IEEE, 2021c, p. 1) ""where all parameters except one are set to their 'base' value and the final parameter is set to one of its other valid values"" (p. 17)","Combinatorial Testing (IEEE, 2022, p. 22; 2021c, pp. 2, 17, Fig. 2), Scenario Testing (can be) (p. 17), Smoke Testing?","Base Value Testing (implied by IEEE, 2021c, pp. 1, 17)","""Generally[,] the base choice is the most 'important' [or ""most frequently used""] value that the parameter can take"" (IEEE, 2021c, p. 17). See Grindal et al., 2005"
"Baseline Testing (Kam, 2008, p. 15)","Technique (implied by Kam, 2008, p. 15)",,,,
Basic Block Testing,Technique (inferred from data flow testing),"Testing that aims to execute ""consecutive parts of code that execute together without any branching"" (Peters and Pedrycz, 2000, p. 477)","Data Flow Testing (Peters and Pedrycz, 2000, pp. 477, 479), Decision Testing (implied by Fig. 12.31)",,
Behavioural Testing,Technique,"The process of ""entering inputs, receiving outputs, and checking the results"" (Patton, 2006, p. 64)","Dynamic Testing, Specification-based Testing (Patton, 2006, pp. 64-65)",,
"Behaviour Analysis? (Gerrard, 2000a, Fig. 4)",Approach,,"Static Testing, W-Model Testing (Gerrard, 2000a, Fig. 4), Static Analysis?",,
Beta Testing,"Level (IEEE, 2022, p. 22; inferred from acceptance testing), Type (implied by Firesmith, 2015, p. 58)","The ""second stage of testing when a product is in limited production use"" (IEEE, 2017, p. 17) ""to determine whether or not … [it] satisfies the user/customer needs and fits within the business processes"" (Kam, 2008, p. 42)","Acceptance Testing (IEEE, 2022, p. 22; Hamburg and Mogyorodi, 2024; Dennis et al., 2012, p. 455), Unscripted Testing (Washizaki, 2024, p. 5-8; implied by Dennis et al., 2012, p. 455), User Testing (implied by Firesmith, 2015, p. 39), Security Testing (Firesmith, 2015, p. 58), Operational Testing (Kam, 2008, p. 42), User as Tester Testing?","Beta Tester Testing, User Testing? (implied by Firesmith, 2015, p. 39)","""Often performed at a customer site"" (IEEE, 2017, p. 45) by ""a larger set of representative users"" (Washizaki, 2024, p. 5-8) or, more specifically, ""at an external site to the developer's test environment by roles outside the development organization"" (Hamburg and Mogyorodi, 2024); related to alpha testing (Washizaki, 2024, p. 5-8)"
Big-Bang Testing,"Technique (Sharma et al., 2021, pp. 601, 603, 605-606), Level (inferred from integration testing)","""Testing in which … [components of a system] are combined all at once into an overall system, rather than in stages"" (IEEE, 2017, p. 45; similar in Peters and Pedrycz, 2000, p. 489)","Integration Testing (IEEE, 2017, p. 45; Washizaki, 2024, p. 5-7; Sharma et al., 2021, p. 603; Kam, 2008, p. 42; Peters and Pedrycz, 2000, p. 488, Tab. 12.8)","Sometimes spelled without a hyphen (Peters and Pedrycz, 2000, p. 489)","""Quite challenging and risky"" (Peters and Pedrycz, 2000, p. 489), but may be less so when done alongside code generation. It makes it relatively easy ""to test paths"" and ""to plan and control"", but these benefits are also provided by bottom-up integration testing (Fig. 12.40)"
"Block Testing (implied by Doğan et al., 2014, Tab. 11)",Technique (inferred from structure-based testing),,"Structure-based Testing (implied by Doğan et al., 2014, Tab. 11), Control Flow Testing?",,
"Blue Team Testing (Firesmith, 2015, p. 57)","Type (implied by Firesmith, 2015, p. 57)",,"Penetration Testing (Firesmith, 2015, p. 57)",,
Bottom-Up (Integration) Testing,"Technique (Sharma et al., 2021, pp. 601, 603, 605-606), Level (inferred from integration testing)","Integration testing that ""starts with the lowest-level components … and proceeds through progressively higher-levels"" (IEEE, 2017, p. 49; OG ISO/IEC 2015; similar in Kam, 2008, p. 42; van Vliet, 2000, p. 438); often uses test drivers (Patton, 2006, p. 109; Peters and Pedrycz, 2000, Tab. 12.8; van Vliet, 2000, p. 438) to ""simulate the environment in which the module being tested is to be integrated"" (van Vliet, 2000, p. 438)","Integration Testing (Washizaki, 2024, p. 5-7; Sharma et al., 2021, pp. 601, 603, 605-606; Kam, 2008, p. 42; Sangwan and LaPlante, 2006, p. 27; Patton, 2006, p. 109; Peters and Pedrycz, 2000, p. 488, Tab. 12.8; van Vliet, 2000, p. 438), Sandwich Integration Testing (Peters and Pedrycz, 2000, Tab. 12.8)",,"Difference between this as integration testing vs. not (as in Firesmith, 2015, p. 28; Gerrard, 2000b, p. 27)? Often used for functional web application testing (Gerrard, 2000b, p. 13)"
"Boundary Condition Testing (Patton, 2006, pp. 70-74, 86; implied by IEEE, 2017, p. 442)",Approach,"Testing using ""error prone"" values ""at the edge of the planned operational limits of the software"", such as ""the valid data just inside the boundary, … the last possible valid data, and … the invalid data just outside the boundary"", that affects different data types, each with its own set of conditions (Patton, 2006, pp. 72-73)","Data Testing, Dynamic Testing, Specification-based Testing, Behavioural Testing (Patton, 2006, pp. 70-74), Boundary Value Analysis, Equivalence Partitioning? (implied by p. 73), Error-based Testing (implied by van Vliet, 2000, p. 399)",,"Boundary conditions should be tested even if they occur “with zero probability”, in case there actually is a case where it can occur; this process of testing may reveal it (Peters and Pedrycz, 2000, p. 460)"
Boundary Value Analysis,"Technique (IEEE, 2022, pp. 2, 20, 22; 2021a, p. 5; 2021c, pp. 1, 7, 12, Fig. 2; Washizaki, 2024, p. 5-11; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 48; Souza et al., 2017, p. 3; Reid, 1996, pp. 2, 7)","Testing ""based on exercising the boundaries of equivalence partitions"" (IEEE, 2022, p. 2; 2021c, p. 1; similar on p. 12 and in Hamburg and Mogyorodi, 2024) (i.e., ""minimum or maximum input[s], internal [values], or output value[s] specified for a system or component"" (IEEE, 2017, p. 49))","Specification-based Testing (IEEE, 2022, pp. 2, 22; 2021c, pp. 1, 12, Fig. 2; Washizaki, 2024, p. 5-11; Hamburg and Mogyorodi, 2024; Sharma et al., 2021, Fig. 1; Souza et al., 2017, p. 3, Reid, 1996, p. 7), Equivalence Partitioning (IEEE, 2022, p. 2; 2021c, p. 1; implied by 2021b, p. 20; 2021c, pp. 10, 12; Hamburg and Mogyorodi, 2024), Domain Analysis (IEEE, 2021c, p. 11; OG Beizer 1995), Model-based Testing (p. 12; implied by 2021b, p. 20), Grey-Box Testing (Firesmith, 2015, p. 48), Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5), Unit Testing (can be) (IEEE, 2021b, p. 55; Reid, 1996, pp. 1-2, 6-7), Negative Testing (can be) (2021c, p. 10; Reid, 1996, p. 7), One-to-One Testing (can be), Minimized Testing (can be) (IEEE, 2021c, pp. 11, 13; OG Myers 1979), Functional Testing? (2022, p. 20)","BVA (Reid, 1996, pp. 1, 7), Boundary Value Testing (Firesmith, 2015, p. 48; Kam, 2008, p. 42), Boundary Testing (implied by IEEE, 2021c, p. 13)","""Contiguous partitions … will result in duplicate test coverage items, in which case it is typical practice to only exercise these duplicated values once"" (IEEE, 2021c, p. 13). Seems to be related to stress testing (IEEE, 2017, p. 49). See Myers (1979); Patton (2006, pp. 70-74)"
Branch Condition Combination Testing,"Technique (IEEE, 2022, p. 22; 2021c, pp. 2, 25, Fig. 2; Hamburg and Mogyorodi, 2024; van Vliet, 2000, p. 422; Kam, 2008, p. 46; Reid, 1996, p. 2)","Testing ""based on exercising [""all possible"" (van Vliet, 2000, p. 422)] combinations of Boolean values of conditions within a decision"" (IEEE, 2021c, p. 2; similar on p. 25; Hamburg and Mogyorodi, 2024; Patton, 2006, p. 120)","Structure-based Testing (IEEE, 2022, p. 22; 2021c, pp. 2, 25, Fig. 2; Hamburg and Mogyorodi, 2024; Sharma et al., 2021, Fig. 1; Kam, 2008, p. 46), Control Flow Testing (IEEE, 2021c, p. 25; Firesmith, 2015, p. 49; Patton, 2006, pp. 120-121), Model-based Testing (IEEE, 2021c, p. 25), Coverage-based Testing (van Vliet, 2000, p. 422), Decision Testing (implied by the caveat of ""atomic conditions"" in Hamburg and Mogyorodi, 2024), Path Testing (implied by Patton, 2006, p. 120), Exhaustive Testing (implied by p. 121), Structural Testing (implied by Reid, 1996, p. 6)","Multiple Condition Testing (Hamburg and Mogyorodi, 2024; van Vliet, 2000, p. 422; Kam, 2008, p. 42), Condition Combination Testing (Hamburg and Mogyorodi, 2024; Kam, 2008, p. 42), Condition Testing (Patton, 2006, p. 120; Sharma et al., 2021, Fig. 1), Extended Branch Coverage (van Vliet, 2000, p. 422), Exhaustive Testing (if ""each subcondition is viewed as a single input"") (Peters and Pedrycz, 2000, p. 464)","Requires a minimum of one test case, even if ""there are no decisions in the test item"" (IEEE, 2021c, pp. 26, 34). ""May be quite challenging"" since ""if each subcondition is viewed as a single input, then this … is analogous to exhaustive testing""; however, there is usually a manageable number of subconditions (Peters and Pedrycz, 2000, p. 464). See Myers (1979)"
Branch Condition Testing,"Technique (IEEE, 2022, p. 22; 2021c, pp. 2, 25, Fig. 2; Hamburg and Mogyorodi, 2024; Washizaki, 2025, p. 5-13; Kam, 2008, p. 43; Reid, 1996, p. 2; implied by Firesmith, 2015, p. 49)","Testing ""based on exercising Boolean values of conditions within decisions and the decision outcomes"" (IEEE, 2021c, p. 2; similar on p. 25)","Structure-based Testing (IEEE, 2022, p. 22; 2021c, pp. 2, 25, Fig. 2; Kam, 2008, p. 43), Control Flow Testing (p. 25; Washizaki, 2025, p. 5-13; implied by Firesmith, 2015, p. 49), Branch Condition Combination Testing (IEEE, 2021c, pp. 25-26), Model-based Testing (p. 25), MC/DC Testing (implied by Reid, 1996, Fig. 2, p. 4), Structural Testing (implied by p. 6)","Condition Testing (Hamburg and Mogyorodi, 2024), Cyclomatic Adequacy Testing? (implied by van Vliet, 2000, pp. 422-423, Fig. 13.17)","Requires a minimum of one test case, even if ""there are no decisions in the test item"" (IEEE, 2021c, pp. 25, 34). ""The use of the term 'condition' … can mislead the reader into thinking all conditions are exercised"" (Reid, 1996, p. 4)"
Branch Testing,"Technique (IEEE, 2022, pp. 2, 22; 2021a, p. 5; 2021c, pp. 2, 8, 23, 125, Fig. 2; Washizaki, 2025, p. 5-13; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 49; Reid, 1996, p. 2)","Testing ""based on exercising branches [or ""execut[ing] each outcome of each decision point"" (IEEE, 2017, p. 51)] in the control flow of the test item"" (IEEE, 2022, p. 2; 2021c, p. 2; similar in van Vliet, 2000, p. 421), which may include conditional/unconditional branches, entry/exit points, and ""sequential statements between an entry and exit point that contains [sic?] no decisions"", but not ""function … [or] method calls"" (IEEE, 2021c, p. 24)","Structure-based Testing (IEEE, 2022, pp. 2, 22; 2021c, pp. 2, 23, 125, Fig. 2; Hamburg and Mogyorodi, 2024), Control Flow Testing (IEEE, 2021c, pp. 2, 23-24, 125-126; Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49; Patton, 2006, pp. 119-120; van Vliet, 2000, p. 421; Reid, 1996, pp. 3-4), Model-based Testing (IEEE, 2021c, pp. 23-24, 125; Reid, 1996, p. 4), Specification-based Testing, Grey-Box Testing (p. 125), Path Testing (Patton, 2006, p. 119; implied by IEEE, 2021c, Fig. F.1; van Vliet, 2000, p. 433; Reid, 1996, p. 3), Branch Condition Combination Testing (Patton, 2006, p. 121; implied by p. 120; van Vliet, 2000, p. 422, Fig. 13.17), Closed-Loop Testing (Preuße et al., 2012, p. 6), Unit Testing (can be to be most effective) (Peters and Pedrycz, 2000, p. 481; OG Miller et al., 1994), Structure-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3), LCSAJ Testing (implied by Reid, 1996, p. 3, Fig. 2), Structural Testing (implied by p. 6), MC/DC Testing (implied by Fig. 2), All-P-Uses Testing (implied by Fig. 2; (only if there are infeasible paths?) van Vliet, 2000, Fig. 13.17), Weak Mutation Testing (implied by Fig. 13.17)","All-Edges Testing (van Vliet, 2000, p. 421), Arc Testing (Kam, 2008, p. 42; implied by IEEE, 2021c, p. 24; Reid, 1996, p. 4), Decision Testing (often) (Reid, 1996, p. 4), Link Testing (implied by IEEE, 2021c, p. 24; Reid, 1996, p. 4), Edge Testing (implied by IEEE, 2021c, p. 24)","Requires a minimum of one test case, even if ""there are no branches in the test item"" (IEEE, 2021c, pp. 24, 33) and at least 85% coverage to be effective (Peters and Pedrycz, 2000, p. 481; OG Miller et al., 1994), although this doesn't guarantee correctness (Patton, 2006, p. 120; van Vliet, 2000, p. 421). ""When testing code where defensive programming techniques have been employed[,] it is unrealistic to expect 100% branch coverage to be achieved"" (Reid, 1996, p. 3)"
Browser Page Testing,Approach,"Testing ""the objects that execute within the browser, but … not … the server-based components"", including code within HTML and applets (Gerrard, 2000b, p. 13)","Smoke Testing, Functional Testing, Dynamic Testing, Desktop Development Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Web Application Testing",,"""Most easily tested against checklists"" (Gerrard, 2000b, p. 13)"
"Buffer Overrun Testing (implied by Patton, 2006, pp. 201-205)",Technique,"Testing that ensures that trying to write data to a destination that is too small does not cause existing data (including code) to be overwritten, which could allow code injection (Patton, 2006, pp. 202, 204-205)","Boundary Condition Testing, Security Testing (Patton, 2006, p. 75), Code Injection? (pp. 202, 204-205)",,"Important in ""languages [sic] such as C and C++, that lack safe string handling functions"" (Patton, 2006, p. 201) where unsafe versions should be replaced with the corresponding safe versions (pp. 203-204). May be supported by fault injection (Ghosh and Voas, 1999, p. 41)"
Bug Hunt Testing,"Technique (Firesmith, 2015, p. 50)",,"Experience-based Testing (Firesmith, 2015, p. 50)",,
Build Verification Testing (BVT),Practice (inferred from automated testing),"""Automated test[ing] that validates the integrity of each new build and verifies its key/core functionality, stability, and testability"" (Hamburg and Mogyorodi, 2024)","Automated Testing, Functionality Testing, Stability Testing, Integrity Testing, (Testability Testing) (Hamburg and Mogyorodi, 2024)",,
"Built-In Testing (Firesmith, 2015, p. 31)",Practice?,,,"BIT (Firesmith, 2015, p. 31)",
"Business Acceptance Testing (Firesmith, 2015, p. 30)","Level (Firesmith, 2015, p. 30; inferred from acceptance testing)",,"Acceptance Testing (Firesmith, 2015, p. 30)","BAT (Firesmith, 2015, p. 30)",
"Business Process-based Testing (Kam, 2008, p. 42)","Technique (Washizaki, 2024, p. 5-12)","Testing ""designed based on descriptions and/or knowledge of business processes"" (Kam, 2008, p. 43), including ""roles in a workflow specification"" (Washizaki, 2024, p. 5-12)","Scenario-based Testing (Washizaki, 2024, p. 5-12)","Business Process Testing (Washizaki, 2024, p. 5-12)",
"Canary (Release) Testing (see Washizaki, 2024, pp. 6-5, 6-10)","Technique (Washizaki, 2024, p. 6-5)","""A partial and time-limited deployment of a change in a service and an evaluation of that change"" to help ""decide whether to proceed with a complete deployment"" (Washizaki, 2024, p. 6-10)",,,"A method for ""testing the software in the production system context"" which ""can be particularly challenging"" (Washizaki, 2024, p. 6-5)"
"Capability Testing (inferred from capability requirements (IEEE, 2024, pp. 14, 22-24))",Approach,"Testing ""of a condition or behavior that a system is to achieve under stated conditions"" (IEEE, 2024, p. 14)","Dependability Testing (if it exists) (IEEE, 2024, p. 22)",,
Capacity Testing,"Type (IEEE, 2022, p. 22; 2013, p. 2; implied by its quality (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1); Firesmith, 2015, p. 53), Technique (IEEE, 2021c, p. 38-39)","Testing that evaluates the ""capability of a product to meet requirements for the maximum limits of a product parameter"", such as the number of concurrent users, transaction throughput, or database size (ISO/IEC, 2023a), or to ""perform under conditions that may need to be supported in the future"", including ""what level of additional resources (e.g. memory, disk capacity, network bandwidth) will be required to support anticipated future loads"" (IEEE, 2021c, p. 39)","Performance Efficiency Testing (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1; 2013, p. 2), Performance-related Testing (2022, p. 22; 2021c, p. 38), Model-based Testing, Requirements-based Testing, Conformance Testing (p. 38), Performance Testing (Washizaki, 2024, p. 5-9; Hamburg and Mogyorodi, 2024; implied by Moghadam, 2019, p. 1187), Backup Testing (can be) (Bas, 2024, pp. 22-23)",,"Has the following subcategories: channel capacity (IEEE, 2017, p. 67), memory capacity (p. 270), storage capacity (p. 441)"
Capture-Replay Driven Testing,"Practice (IEEE, 2022, Fig. 2; inferred from automated testing)","""A test automation approach in which inputs to the test object are recorded during manual testing to generate automated test scripts that can be executed later"" (Hamburg and Mogyorodi, 2024)","Automated Testing (IEEE, 2022, pp. 22, 35; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 44), Specification-based Testing (implied by Doğan et al., 2014, p. 194)","Capture/Playback, Capture/Replay, Record/Playback (Hamburg and Mogyorodi, 2024), Record-Playback Testing? (Firesmith, 2015, p. 44)",
Cause-Effect Graphing,"Technique (IEEE, 2022, pp. 2, 22; 2021c, pp. 2, 18, Fig. 2; Washizaki, 2024, p. 5-11; Reid, 1996, p. 2; implied by Firesmith, 2015, p. 47)","Testing ""based on exercising decision rules [""between causes … and effects"" (IEEE, 2021c, p. 18)] in a cause-effect graph"" (2022, p. 2; 2021c, p. 2)","Specification-based Testing (IEEE, 2022, p. 22; 2021c, pp. 2, 18, Fig. 2; Washizaki, 2024, p. 5-11; Firesmith, 2015, p. 47; Peters and Pedrycz, 2000, Tab. 12.1, Fig. 12.2), Model-based Testing (IEEE, 2021c, p. 18), Dynamic Testing (Peters and Pedrycz, 2000, Tab. 12.1; Gerrard, 2000a, Fig. 5), W-Model Testing (Fig. 5)","Cause and Effect Testing? (Firesmith, 2015, p. 47)","Related to decision table testing (Washizaki, 2024, p. 5-11; implied by IEEE, 2021c, pp. 18-19). See Myers (1979); Nursimulu and Probert (1995)"
Certification,Approach,"The ""process of confirming that a system or component complies with its specified requirements and is acceptable for operational use"" (IEEE, 2017, p. 63; similar in Hamburg and Mogyorodi, 2024; Washizaki, 2024, p. 5-4) that is usually done by a third-party (IEEE, 2017, p. 63; OG ISO/IEC 2015)","Role-based Testing? (usually) (IEEE, 2017, p. 63; OG ISO/IEC 2015)",,
CGI Component Testing,Approach,"Testing that ""covers the objects that execute on the server, but are initiated by forms-based user interactions on the browser"" (Gerrard, 2000b, p. 13)","Smoke Testing, Infrastructure Testing, Functional Testing, Dynamic Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Transaction Testing, Unit Testing (implied by p. 16)","Common Gateway Interface Component Testing (Gerrard, 2000a, p. 17)","If the UI is ""the last thing to be written"", ""server-based components cannot be tested completely until very late in the project"" (Gerrard, 2000b, p. 13), so test drivers may be used (p. 14). The functional testing portion of this is often developer testing (p. 14).  ""More effective and economic"" when automated due to the potential for ""complex transactions that interface with legacy systems"" (p. 13)"
Change-Related Testing,Approach,"""Testing initiated by modification to a component or system"" (Hamburg and Mogyorodi, 2024)",,,
"Checked Statement Testing (Sakamoto et al., 2013, p. 356; OG Schuler and Zeller, 2011)",Technique (inferred from statement testing),,"Statement Testing, Structure-based Testing (Sakamoto et al., 2013, p. 356; OG Schuler and Zeller, 2011)",,"See Schuler and Zeller, 2011"
Checklist-based Testing,"Practice (IEEE, 2022, p. 34), Technique (Hamburg and Mogyorodi, 2024)","Testing generated ""based on a list of pre-determined items"" that ""can be based on personal experience, commonly found defects, and perceived risks"" (IEEE, 2022, p. 34) or consist of ""questions or required attributes"" (Hamburg and Mogyorodi, 2024)","Experience-based Testing (IEEE, 2022, p. 34; Hamburg and Mogyorodi, 2024)","Checklist Analysis? (IEEE, 2017, p. 67)","""Often focused on a particular quality characteristic"" (IEEE, 2022, p. 34) and can be performed on more than just code; see Patton's ""Specification Terminology Checklist"" (2006, p. 61) and (pp. 99-103)"
Classification Tree Method,"Technique (IEEE, 2022, p. 22; 2021c, pp. 2, 12, Fig. 2; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 47)","Testing ""based on exercising classes in a classification tree"" (IEEE, 2021c, p. 2)","Specification-based Testing (IEEE, 2022, p. 22; 2021c, pp. 2, 12, Fig. 2; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 47), Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, p. 3; 2021c, pp. 6, 12)","Classification Tree Technique (Hamburg and Mogyorodi, 2024), Classification Tree Testing (Firesmith, 2015, p. 47)","Can be done based on ""minimality"" by covering ""the classes represented by leaf nodes"" or on ""maximality"" by covering ""unique combinations of the classes represented by leaf nodes"" (IEEE, 2021c, p. 12). Related to combinatorial testing (Hamburg and Mogyorodi, 2024) and equivalence partitioning (although with disjoint partitions) and can be done recursively (IEEE, 2021c, p. 12). See Grochtmann and Grimm 1993"
CLI Testing,Approach,"""Testing performed by submitting commands to the software under test using a dedicated command-line interface"" (Hamburg and Mogyorodi, 2024)",,,
"Closed Beta Testing (Firesmith, 2015, p. 58)","Type (implied by Firesmith, 2015, p. 58), Level (inferred from beta testing)",,"Beta Testing (Firesmith, 2015, pp. 39, 58)",,
Closed Loop Testing (Control Flow),Technique?,"Loop testing performed on loops with ""no exit … whose execution can be interrupted only by intervention from outside the computer program or procedure in which the [it] is located"" (IEEE, 2017, p. 71)",Loop Testing,,"Not explicitly described by a source (in the context of software control flows); implied by loop testing (Gerrard, 2000a, Fig. 5) and closed loops (IEEE, 2017, p. 71)"
Closed Loop Testing (Control Systems),Technique?,"Testing that ""evaluate[s] the performance of the physical device(s) under actual operating conditions prior to their installation in the real network"" (Forsyth et al., 2004, p. 329)","Specification-based Testing (Preuße et al., 2012, pp. 5-6), Formal Testing (p. 6), Control System Testing? (Forsyth et al., 2004, pp. 329, 331), Performance Testing (can be) (Preuße et al., 2012, p. 2; Forsyth et al., 2004, pp. 329, 331), Correctness Testing (can be) (Preuße et al., 2012, pp. 4, 6), Functional Testing (can be), Non-functional Testing (can be) (p. 6), Online Testing (can be) (Forsyth et al., 2004, p. 329), Model-based Testing (implied by Preuße et al., 2012), Domain-specific Testing?","Sometimes spelled with a hyphen (Preuße et al.), Closed-Loop Verification? (Preuße et al., 2012, p. 1)","This definition seems exclusive to hardware, but may be generalizable. ""The only way to do [sic] get a comprehensive assertion about the correctness of control software"" since the state space is restricted to ""practically relevant"" inputs (Preuße et al., 2012, p. 4). Often performed using ""real time simulators"" (Forsyth et al., 2004, p. 332). See Trudnowski et al., (2017; if in scope)"
"Cloud Testing (Firesmith, 2015, p. 42)",Approach,,"Backup and Recovery Testing (can be?) (implied by Bas, 2024, p. 26)",,
Code Injection,Approach,"""A type of security attack performed by inserting malicious code at an interface into an application to exploit poor handling of untrusted data"" (Hamburg and Mogyorodi, 2024)","Security Attacks (Hamburg and Mogyorodi, 2024), Interface Testing (implied by Hamburg and Mogyorodi, 2024)",,
"(Stepwise) Code Reading (Sharma et al., 2021, p. 601; OG [3, 4, 5])","Approach (Sharma et al., 2021, p. 601; OG [3, 4, 5])",,,"Grey-Box Testing (implied by Sharma et al., 2021, p. 601; OG [3, 4, 5])",
Code Reviews,Approach,"A ""meeting at which software code is presented to project personnel, managers, users, customers, or other interested parties for comment or approval"" (IEEE, 2017, p. 74); can also be done ""using a pull request technique/tool … before [the code] can be merged into [the] project"" (Washizaki, 2024, p. 12-13)","Reviews, Peer Reviews (sometimes), Static Analysis? (Washizaki, 2024, p. 12-13)",,"Sometimes kept separate from testing (Washizaki, 2024, p. 12-11). Seems to line up most closely with the notion of ""peer review"" from Patton (2006, p. 94) and van Vliet (2000, p. 414)"
Co-existence Testing,"Type (implied by its quality (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1); inferred from compatibility and interoperability testing)","""Testing that measures the degree to which a test item can function satisfactorily alongside other independent products in a shared environment"" (IEEE, 2022, p. 3; similar in ISO/IEC, 2023a) ""without a negative impact on any of them"" (Hamburg and Mogyorodi, 2024; similar in ISO/IEC, 2023a)","Compatibility Testing (ISO/IEC, 2023a; IEEE, 2022, p. 3; 2021c, Tab. A.1)","Sometimes spelled without a hyphen (Hamburg and Mogyorodi, 2024)","test approach not mentioned explicitly by IEEE (2022), Hamburg and Mogyorodi (2024), or ISO/IEC (2023a). Related to portability testing (Hamburg and Mogyorodi, 2024)"
Combinatorial Testing,"Technique (IEEE, 2022, pp. 3, 22; 2021b, p. 58; 2021c, pp. 2, 15, Fig. 2; Washizaki, 2024, p. 5-11; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 47; Engström and Petersen, 2015, p. 1)","A ""class of … techniques"" that ""systematically derive a meaningful and manageable [(i.e., finite)] subset of test cases … in terms of test item parameters and the values these parameters can take"" (P-V pairs) (IEEE, 2021c, p. 15)","Specification-based Testing (IEEE, 2022, pp. 3, 22; 2021c, pp. 2, 15, Fig. 2; Washizaki, 2024, p. 5-11; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 47), Model-based Testing (IEEE, 2022, p. 13; 2021c, pp. 6, 15), Mathematical-based Testing (can be) (2022, p. 36), Sampling (implied by p. 18)",,"""Other test design techniques, such as equivalence partitioning or boundary value analysis, … [can be used to] reduce the large set of possible values for a parameter to a manageable subset"" (IEEE, 2021c, p. 16). Related to the classification tree method (Hamburg and Mogyorodi, 2024). See Grindal et al., 2005"
"Command-Form Testing (Doğan et al., 2014, Tab. 13; OG Halfond and Orso, 2006; OG 2007; OG Halfond et al., 2009)",Technique (inferred from database coverage testing),"Testing that aims to ""adequately exercis[e] the interactions between an application and its underlying database"" (Doğan et al., 2014, Tab. 13; OG Halfond and Orso, 2006; OG 2007; OG Halfond et al., 2009)","Web Application Testing, Database Coverage Testing? (Doğan et al., 2014, Tab. 13), Integration Testing?",,
"Comparison Testing (Sharma et al., 2021, p. 601; OG [10])",Approach,,,,
Compatibility Testing,"Type (IEEE, 2022, pp. 3, 22; 2021c, p. 37, Tab. A.1; 2013, p. 2; implied by its quality (ISO/IEC, 2023a); Firesmith, 2015, p. 53)","""Testing that measures the degree to which a test item can function satisfactorily alongside other independent products in a shared environment (co-existence), and where necessary, exchanges information with other systems or components (interoperability)"" (IEEE, 2022, p. 3; 2013, p. 2; similar in ISO/IEC, 2023a and Hamburg and Mogyorodi, 2024)","Requirements-based Testing (IEEE, 2021c, p. 37), Non-functional Testing (Washizaki, 2025, p. 5-9; Bertolino et al., 2019, p. 93:11), Regression Testing (can be) (Washizaki, 2024, p. 5-8)","Co-existence Testing (incorrectly) (IEEE, 2021c, p. 37)","Not atomic and implies the existence of ""co-existence testing"" and ""interoperability testing"", which is made more explicit by ISO/IEC (2023a). Includes both ""different hardware and software facilities"" and ""different versions or releases"" (Washizaki, 2024, p. 5-9) and may be ""applied to multiple copies of the same test item or to multiple test items sharing a common environment"" (IEEE, 2021c, p. 37). Has subcomponents: ""order of installation"", ""order of instantiation"", ""concurrent use"", and ""environment constraints"" (IEEE, 2021c, p. 37), as well as potentially ""downward compatibility"" (implied by IEEE, 2017, p. 147) and ""upward compatibility"" (implied by p. 492)"
"Compiler Testing (Mandl, 1985)",Approach,,,,
"Compiler-based Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Propagation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",,
"Complete Regression Testing (Firesmith, 2015, p. 34)",Approach,,"Regression Testing (Firesmith, 2015, p. 34)",,
Compliance Testing,"Type (implied by its quality (IEEE, 2017, p. 82; Hamburg and Mogyorodi, 2024))","Testing that ""validates"" (Firesmith, 2015, p. 33) the ""adherance of [the test item] … to standards, conventions or regulations in laws and similar prescriptions"" (Hamburg and Mogyorodi, 2024; OG IREB Glossary; similar in Firesmith, 2015, p. 33) usually ""forced by an external regulatory body"" (Washizaki, 2024, p. 5-8)","Security Testing (Bas, 2024, p. 28)","Regulatory-Compliance Testing (Firesmith, 2015, p. 33), Regulation Testing (Kam, 2008, p. 47), Standards Testing (p. 48)","""Audits are often used to verify compliance with standards"" (IEEE, 2022, p. 44)"
Component Integration Testing,"Level (IEEE, 2017, p. 467; Hamburg and Mogyorodi, 2024; Black, 2009, pp. 14, 18; inferred from integration testing)","""Testing of groups of related components"" (IEEE, 2017, p. 82) or ""integration testing of components"" (Hamburg and Mogyorodi, 2024) ""performed to expose defects in the interfaces and interaction between [them]"" (Kam, 2008, p. 43)","Integration Testing (Hamburg and Mogyorodi, 2024)","Module Integration Testing, Unit Integration Testing (Hamburg and Mogyorodi, 2024), Link Testing (Kam, 2008, p. 45)",
"Computation Flow Testing (implied by Peters and Pedrycz, 2000, Tab. 12.2)",Approach,"Testing where ""a program is viewed as a collection of computations treated as a trace of data states produced in response to a particular input"" (Peters and Pedrycz, 2000, Tab. 12.2)","Data Flow Testing?, Computation Testing?",Computation Testing?,"Can be supported by ""fault seeding, mutation analysis, and sensitivity analysis"" (Peters and Pedrycz, 2000, Tab. 12.2)"
"Computation Testing (Peters and Pedrycz, 2000, Tab. 12.1)",Approach,,"Structure-based Testing, Dynamic Testing (Peters and Pedrycz, 2000, Tab. 12.1)",,
"Concrete Execution (Doğan et al., 2014, p. 192)",Technique (inferred from symbolic execution),,Static Testing?,,"Related to symbolic execution (Doğan et al., 2014, p. 192)"
Concurrency Testing,Approach,"""Testing to determine how the occurrence of two or more activities within the same interval of time, achieved either by interleaving the activities or by simultaneous execution, is handled"" (Kam, 2008, p. 43)","Performance Testing (Gerrard, 2000b, p. 23), Compatibility Testing?, Requirements-based Testing? (IEEE, 2021c, p. 37)",,"Can find ""obscure bugs"" ""over extended periods"" (Gerrard, 2000b, p. 23). May apply to ""two or more test items … perform[ing] their required functions while running (but not necessarily communicating) in the same environment"" (IEEE, 2021c, p. 37)?"
"Conditional Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Infection-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",,
Configuration Testing,"Type (implied by Firesmith, 2015, p. 53)","Testing that ""verifies the software under specified configurations"" (Washizaki, 2024, p. 5-10), or ""arrangement[s] … defined by … its constituent parts[,] … requirements, design, and[/or] implementation"" (IEEE, 2017, p. 90), when it ""is built to serve different users"" (Washizaki, 2024, p. 5-10)","Non-functional Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1, p. 20), Smoke Testing, Dynamic Testing (2000a, Tab. 2; 2000b, Tab. 1), Data Center Testing (Firesmith, 2015, p. 24), Combinatorial Testing (can be) (IEEE, 2021c, p. 15), Developer Testing (can be) (Gerrard, 2000a, p. 11)","Portability Testing (Kam, 2008, p. 43)","Can be done by an audit (IEEE, 2017, p. 90). For web application testing, this includes combinations of OS platforms, network connections, commercial services, and browsers (Gerrard, 2000b, p. 20)"
Conformance Testing,"Type (implied by its use of quality requirements (PMBOK, 2013, p. 557))","Testing that evaluates the degree to which ""results … fall within the limits that define acceptable variation for a quality requirement"" (PMBOK, 2013, p. 532) ""to obtain the conviction that its [the SUT's] behaviour conforms with its specification"" (Jard et al., 1999, p. 25) ""in its test environment"" (p. 26; OG [4]) or that ""aims to verify that the SUT conforms to standards, rules, specifications, requirements, design, processes, or practices"" (Washizaki, 2025, p. 5-7)","Quality Assurance (PMBOK, 2013, Fig. 8-2), Formal Testing (can be) (Jard et al., 1999, pp. 25-26), SoS Testing (can be) (IEEE, 2019b, p. 18), Requirements-based Testing (implied by PMBOK, 2013, pp. 229, 235)","Functional Testing (Washizaki, 2025, p. 5-7; implied by IEEE, 2017, p. 93), Correctness Testing (Washizaki, 2025, p. 5-7), Compliance Testing (although this may be wrong) (Kam, 2008, p. 43)","May be contrasted with nonconformity (IEEE, 2017, p. 292)"
Consistency Testing,"Type (implied by Firesmith, 2015, p. 53)","Testing the ""degree of uniformity, standardization, and freedom from contradiction"" (IEEE, 2017, p. 94)",,,"Content consistency testing is possibly a subapproach (see IEEE, 2017, p. 96). Related to traceability testing (p. 94)"
Construction Testing,Approach,"Testing that ""aims to reduce the gap between when faults are inserted into the code and when those faults are detected, thereby reducing the cost incurred to fix them"" (Washizaki, 2025, p. 4-7)","Maintenance Testing (Washizaki, 2025, p. 4-1)",,"""Often performed by the software engineer who wrote the code"" and ""does not typically include … more specialized testing"" (Washizaki, 2025, p. 4-7). IEEE (2017, p. 95) defines a process just called ""construction"" that includes unit testing (but not integration testing). See IEEE Standard 829-1998 and IEEE Standard 1008-1987"
Content Checking,Approach,"Testing ""the content of Web pages … [for] accuracy, completeness, consistency, spelling[, and] … accessibility"" (Gerrard, 2000a, p. 3)","Usability Testing, Static Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1, p. 3), Desktop Development Testing (2000a, Tab. 2; 2000b, Tab. 1), Web Application Testing, Grey-Box Testing? (Patton, 2006, p. 220)",,Difference between this and content usage testing?
"Content Usage Testing (Firesmith, 2015, p. 58)","Type (implied by Firesmith, 2015, p. 58)",,"Usability Testing (Firesmith, 2015, p. 58)",,Difference between this and content checking?
Continuity Testing,"Type (Koomen et al., 2006, p. 50)",,"Non-functional Testing (Washizaki, 2025, p. 6-8)",,"Given as a type of hardware testing (Firesmith, 2015, p. 21), which would make this an out-of-scope test approach "
Continuous Testing,"Practice (Washizaki, 2024, p. 6-13)","Testing ""started via an automated process that can occur on-demand"" (IEEE, 2022, p. 35) ""that involves testing the software at every stage of the software development life cycle"" (Washizaki, 2025, p. 6-13) ""to obtain feedback … as rapidly as possible"" (Hamburg and Mogyorodi, 2024)","Automated Testing (IEEE, 2022, p. 35; Hamburg and Mogyorodi, 2024), Lifecycle-based Testing (Firesmith, 2015, p. 29), Developer Testing (can be) (Gerrard, 2000a, p. 11)","CT (Firesmith, 2015, p. 29)","Involves testing early (IEEE, 2022, p. 35; Washizaki, 2025, p. 6-13; Hamburg and Mogyorodi, 2024), often (Washizaki, 2025, p. 6-13; Hamburg and Mogyorodi, 2024), and everywhere (Hamburg and Mogyorodi, 2024). Difficult to do alongside ""geographically distributed development"" (Sangwan and LaPlante, 2006, p. 27)"
Contractual Acceptance Testing,"Level (Firesmith, 2015, p. 30; inferred from acceptance testing)","""Acceptance testing performed to verify whether a system satisfies its contractual requirements"" (Hamburg and Mogyorodi, 2024)","Acceptance Testing (Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 30)","Contract Acceptance Testing, CAT (conflicts with ""customer AT"") (Firesmith, 2015, p. 30)",
Control Flow Analysis,Approach,"""Analysis based on a representation of unique paths for executing a component or system"" (Hamburg and Mogyorodi, 2024)","Static Analysis (Hamburg and Mogyorodi, 2024)",,
Control Flow Testing,"Technique (Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49; Souza et al., 2017, p. 3)","Testing that covers all of a particular subdivision of code (Washizaki, 2025, p. 5-13; van Vliet, 2000, p. 421), namely the ""sequence in which operations are performed during … execution"" (IEEE, 2017, p. 101; OG IEEE, 2015; similar in Peters and Pedrycz, 2000, Tab. 12.2)","Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, pp. 3, 20; 2021c, pp. 6, 10; implied by Doğan et al., 2014, p. 179), Structure-based Testing (Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49; Lyu, 1996, pp. 532-533; Patton, 2006, p. 117; Souza et al., 2017, p. 3), Dynamic Testing (p. 117), Web Application Testing (with its subtechniques) (Doğan et al., 2014, p. 179), Coverage-based Testing (implied by van Vliet, 2000, pp. 420-421)","Code Coverage Testing (Patton, 2006, pp. 117-121)","""Complete coverage … is a praiseworthy goal, but … [is] not always achievable, or perhaps desirable"" and infeasible code may be excluded from coverage calculations with adequate justification and documentation (Reid, 1996, p. 3). ""The adequacy of such tests is measured in percentages"" and can also be performed on ""blocks of statements[] or specific combinations of statements"" (Washizaki, 2025, p. 5-13); would these be a different testing type? Related to data flow testing? (IEEE, 2017, p. 101)"
"Control System Testing (Forsyth et al., 2004)",Approach,"Testing of ""system[s] which respond[] to input signals from parts of machine elements, operators, external control equipment or any combination of these and generate[] output signals causing the machine to behave in the intended manner"" (ISO, 2015)","System Testing, Domain-specific Testing?","Machine Control System Testing (implied by ISO, 2022)","Often performed using ""real time simulators"", which requires ""large amounts of flexible, accurate, high speed I/O"" (Forsyth et al., 2004, pp. 332-333). Can be performed ""at the manufacturer's facility [or] … where the equipment is integrated in the actual power system"" (p. 333)"
Conversion Testing,"Type (IEEE, 2022, p. 22; 2021c, p. 37, Tab. A.1)","Testing ""to determine whether data or software can continue to provide required capabilities after modifications are made to their format, such as converting a program from one programming language to another[,] … converting a flat data file or database from one format to another"" (IEEE, 2021c, p. 37), or ""in replacement systems"" (Kam, 2008, p. 43)","Model-based Testing, Requirements-based Testing, Conformance Testing (IEEE, 2021c, p. 37), Functional Suitability Testing (Tab. A.1), Regression Testing (can be) (Washizaki, 2025, p. 5-8), Portability Testing?","Migration Testing (Kam, 2008, p. 46)",
Cookie Testing,Approach,"Testing based on the use of cookies to ""get [a]round the stateless nature of the web and how this might be subverted"" (Gerrard, 2000b, p. 18), such as by ""crack[ing] the password algorithm"" (pp. 28-29)","Penetration Testing (Gerrard, 2000b, p. 28), Web Application Testing",,
Correctness Testing,"Type (implied by its quality (IEEE, 2017, p. 104; Washizaki, 2024, p. 3-13); Firesmith, 2015, p. 53)","Testing the ""degree to which a system or component is free from faults in its specification, design, and implementation[,] … meet[s] specified requirements[, or] … meet[s] user needs and expectations"" (IEEE, 2017, p. 104)","Functional Suitability Testing (ISO/IEC, 2023a)","Conformance Testing, Functional Testing (Washizaki, 2025, p. 5-7)","""Proving the correctness of software … applies only in circumstances where software requirements are stated formally"" and assumes ""these formal requirements are themselves correct"" (van Vliet, 2000, p. 398)"
"COTS Testing (Firesmith, 2015, p. 34)",Approach,,"Reuse Testing (Firesmith, 2015, p. 34)","Commercial Off-the-Shelf Testing (Hamburg and Mogyorodi, 2024)",
"COTS Vendor Testing (Firesmith, 2015, p. 37)",Practice?,,"Development Organization Testing (Firesmith, 2015, p. 37), COTS Testing",,
Coverage-based Testing,"Technique (van Vliet, 2000, p. 398; Souza et al., 2017, p. 3)","Testing focused on ""the coverage of the product to be tested"", often based ""on the notion of a control graph"" (van Vliet, 2000, p. 420), by ""test[ing] the program's states and the program's flow among them"" (Patton, 2006, p. 117)","Structure-based Testing (Souza et al., 2017, p. 3), Requirements-based Testing (can be) (van Vliet, 2000, p. 425)",,"May be measured using a cyclomatic complexity metric (van Vliet, 2000, p. 422) and/or performed on the specification by ""depict[ing the requirements] as a graph, where the nodes denote elementary requirements and the edges denote relations between [them]"", but it can be difficult to assess since the specific data available in each node are not apparent (pp. 425-426). Allows for redundant and/or missing test cases to be identified and can be supported by code coverage analyzers (Patton, 2006, pp. 117-118); generating infrastructure for this analysis may be a worthwhile goal in Drasil (since we know the structure of the generated code, to some extent, beforehand), but not all techniques are feasible to accomplish completely (e.g., path testing)"
"Cross-Browser Compatibility Testing (Doğan et al., 2014, Tab. 8)",Approach,"Testing the ""appearance and behavior"" of a website ""in different browsers"" (Choudhary et al., 2010, p. 1)","Web Application Testing (Doğan et al., 2014, Tabs. 8, 22), UI Testing, Functionality Testing (Choudhary et al., 2010, p. 1), Automated Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1; Choudhary et al., 2010, p. 3), Back-to-Back Testing, Differential Assertion Checking (implied by p. 1), Regression Testing (can be) (Tab. 1), Manual Testing (often) (pp. 1, 5-6), Developer Testing (Gerrard, 2000a, p. 11), Fault-based Testing (implied by Doğan et al., 2014, pp. 195-196), Configuration Testing (implied by Gerrard, 2000a, p. 11), Compatibility Testing","Cross-Browser Compliance Testing (implied by Choudhary et al., 2010, p. 1), Browser Syntax Compatibility (Testing?) (implied by Gerrard, 2000b, p. 5)","""Issues range from simple cosmetic problems in the user interface to critical functionality failures"" and ""are observed on the client side"" (Choudhary et al., 2010, p. 1). Can be performed based on structural and/or visual analysis (p. 2) and using multiple platforms, browsers, and/or version (p. 3)"
Crowd Testing,"Approach (Hamburg and Mogyorodi, 2024)","Testing performed by ""a large group of testers"" (Hamburg and Mogyorodi, 2024)",,,
Customer Acceptance Testing,"Level (Firesmith, 2015, p. 30; inferred from acceptance testing)",,"Acceptance Testing (Firesmith, 2015, p. 30)","CAT (conflicts with ""contract(ual) AT"") (Firesmith, 2015, p. 30)",
Customer Product Integration Testing,"Level (Black, 2009, p. 11)",,Integration Testing,"Usability Testing (implied by Washizaki, 2025, p. 5-10), Customer Acceptance Testing",
Dark Launches,"Technique (Washizaki, 2024, p. 6-5)",,,,"A method for ""testing the software in the production system context"" which ""can be particularly challenging"" (Washizaki, 2024, p. 6-5)"
"Data Center Testing (Firesmith, 2015, p. 24)",Level (inferred from operational testing),,"Operational Testing (Firesmith, 2015, p. 30)",,
"Data Dependence Transition Relation Testing (Doğan et al., 2014, Tab. 13; OG Peng and Lu, 2011)",Technique (inferred from coverage-based testing),,"Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13), Data-driven Testing?",,
Data Flow Analysis,Approach,"""Analysis based on the lifecycle of variables"" (Hamburg and Mogyorodi, 2024)","Static Analysis (Hamburg and Mogyorodi, 2024)",,
Data Flow Testing,"Technique (IEEE, 2022, p. 22; 2021c, pp. 3, 27, Fig. 2; Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49; Souza et al., 2017, p. 3; Kam, 2008, p. 43; Reid, 1996, p. 2; implied by Peters and Pedrycz, 2000, pp. 476-481)","Control flow testing ""based on exercising definition-use pairs"" (IEEE, 2021c, p. 3; similar on p. 27; Lyu, 1996, p. 533) that requires additional ""information about how the variables are defined, used, and killed"" (Washizaki, 2025, p. 5-13; similar in Peters and Pedrycz, 2000, Tab. 12.2) ","Structure-based Testing (IEEE, 2022, p. 22; 2021c, pp. 3, 27, Fig. 2; Lyu, 1996, pp. 532-533; Souza et al., 2017, p. 3; Kam, 2008, p. 43), Control Flow Testing (IEEE, 2021c, p. 27; implied by Washizaki, 2025, p. 5-13; IEEE, 2017, p. 101), Model-based Testing (2021c, p. 27; implied by Doğan et al., 2014, p. 179), Exhaustive Testing (Patton, 2006, p. 121; implied by Peters and Pedrycz, 2000, p. 467), Propagation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3), Web Application Testing (Doğan et al., 2014, p. 179; can be in Kam, 2008, pp. 16-17), Mutation Testing (likely) (Lyu, 1996, p. 534; OG Math91 and Wong93), Path Testing (implied by Lyu, 1996, p. 533)",,"Many anomalies can result from the three possible actions on data: defining, killing (occurs ""when the data cease to have a state""), and using (Peters and Pedrycz, 2000, pp. 478, 480; OG Beizer, 1990). May be supported by debugger tools to check or modify the values of variables (Patton, 2006, p. 114) and/or involve data flow analysis or ""data analysis"". See Burnstein (2003) and Horg90"
"Data Generation (Testing) (Peters and Pedrycz, 2000, Tab. 12.1)",Approach,,"Structure-based Testing, Dynamic Testing (Peters and Pedrycz, 2000, Tab. 12.1)",,
Data Integrity Testing,Approach,"""Testing the methods and processes used to access and manage the data and to ensure access methods, processes and data rules function as expected"" (Kam, 2008, p. 44)","Security Testing (Bas, 2024, p. 28), Integrity Testing",,
"Data Migration Testing (Firesmith, 2015, p. 53)","Type (IEEE, 2021c, p. 37; implied by Firesmith, 2015, p. 53)",,"Conversion Testing (IEEE, 2021c, p. 37), Correctness Testing (Firesmith, 2015, p. 53), Data Integrity Testing?, Database Admin Testing?",,
Data Testing,Technique,"The process of ""checking that information the user inputs [and] results"", both final and intermediate, ""are handled correctly"" (Patton, 2006, p. 70)","Dynamic Testing (Patton, 2006, p. 70), Structure-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Data-driven Testing?,
"Database Admin Testing (Firesmith, 2015, p. 39)",Practice?,,"Operator Testing (Firesmith, 2015, p. 39)",,
"Database Testing (implied by Doğan et al., 2014, Tab. 13; OG Alalfi et al., 2010)",Technique (inferred from coverage-based testing),,"Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13), Control Flow Testing?",,"Includes page access, SQL statement, and server environment variable testing (Doğan et al., 2014, Tab. 13; OG Alalfi et al., 2010)"
Database Integrity Testing,Approach,"Data integrity testing where data is stored in a database that also ensures ""that during access to the database, data is not corrupted or unexpectedly deleted, updated or created"" (Kam, 2008, p. 44)","Web Application Testing (implied by Kam, 2008, p. 9), Backup Testing (can be) (implied by Bas, 2024, p. 25), Integrity Testing, Data Integrity Testing?",,
Data-driven Testing,"Practice (IEEE, 2022, p. 22), Technique (Kam, 2008, p. 43; OG Fewster and Graham)","Testing ""that uses data files to contain the test data [including ""test input"" (Kam, 2008, p. 43)] and expected results needed to execute the test scripts"" (Hamburg and Mogyorodi, 2024)","Automated Testing (IEEE, 2022, pp. 22, 35; Firesmith, 2015, p. 44; implied by Hamburg and Mogyorodi, 2024), Scripted Testing (Kam, 2008, p. 43)","Sometimes spelled without a hyphen (Kam, 2008, p. 43)","Similar to keyword-driven testing (Hamburg and Mogyorodi, 2024). Can be used to support ""test execution tools such as capture/playback tools"" (Kam, 2008, p. 43)"
Decision Condition Testing,"Technique (Kam, 2008, p. 44)","Testing ""in which test cases are designed to execute condition outcomes and decision outcomes"" (Kam, 2008, p. 44)","Specification-based Testing (Kam, 2008, p. 44), MC/DC Testing?","Condition/Decision Testing? (Washizaki, 2025, p. 5-13)",
Decision Table Testing,"Technique (IEEE, 2022, pp. 4, 22; 2021a, p. 5; 2021c, pp. 3, 18, Fig. 2; Washizaki, 2024, p. 5-11; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 47)","Testing ""based on exercising decision rules in a decision table"" (IEEE, 2022, p. 4; 2021c, p. 3; similar on p. 18 and in Hamburg and Mogyorodi, 2024; Peters and Pedrycz, 2000, pp. 448, 450) ""by considering every possible combination of conditions and their corresponding resultant actions"" (Washizaki, 2024, p. 5-11; similar in Peters and Pedrycz, 2000, p. 450)","Specification-based Testing (IEEE, 2022, pp. 4, 22; 2021c, pp. 3, 18, Fig. 2; Washizaki, 2024, p. 5-11; Hamburg and Mogyorodi, 2024; Sharma et al., 2021, Fig. 1; Firesmith, 2015, p. 47; Peters and Pedrycz, 2000, Fig. 12.2), Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, pp. 3, 20; 2021c, pp. 6, 18; Bourque and Fairley, 2014, p. 4-10), Data Testing (Peters and Pedrycz, 2000, pp. 448, 450-453)","Decision Table-based Testing (Peters and Pedrycz, 2000, pp. 448, 450-453)","Any ""infeasible decision rules … do not need to be covered during testing"" (IEEE, 2021c, p. 18). Related to cause-effect graphing (Washizaki, 2024, p. 5-11). See Myers (1979)"
Decision Testing,"Technique (IEEE, 2022, pp. 4, 22; 2021c, pp. 3, 24, Fig. 2; Washizaki, 2025, p. 5-13; Hamburg and Mogyorodi, 2024; Reid, 1996, p. 2)","Testing ""based on exercising decision outcomes in the control flow of the test item"" (IEEE, 2022, p. 4; 2021c, p. 3; similar on p. 24; Hamburg and Mogyorodi, 2024; Lyu, 1996, p. 533)","Structure-based Testing (IEEE, 2022, pp. 4, 22; 2021c, pp. 3, 24, Fig. 2; Hamburg and Mogyorodi, 2024; Lyu, 1996, p. 533; Sharma et al., 2021, Fig. 1; can be in Reid, 1996, p. 4), Control Flow Testing (IEEE, 2022, p. 4; 2021c, pp. 3, 24; Washizaki, 2025, p. 5-13; can be in Reid, 1996, p. 4), Model-based Testing (IEEE, 2021c, p.  24; can be in Reid, 1996, p. 4), All-P-Uses Testing, All-C-Uses Testing, (Lyu, 1996, p. 534; OG Clar89), Data Flow Testing (Lyu, 1996, p. 534), Peters and Pedrycz, 2000, Fig. 12.13), Path Testing (implied by IEEE, 2021c, Fig. F.1; Lyu, 1996, p. 533), All-P-Uses Testing (implied by IEEE, 2021c, Fig. F.1; Peters and Pedrycz, 2000, Fig. 12.31), Structural Testing (implied by IEEE, 2021c, Fig. F.1), All-C-Uses Testing (implied by Peters and Pedrycz, 2000, Fig. 12.31), Decision Condition Testing (implied by Kam, 2008, p. 44)","Condition Testing (implied by Washizaki, 2025, p. 5-13), Branch Testing (often) (Reid, 1996, p. 4)","Requires a minimum of one test case, even if ""there are no decisions in the test model"" (IEEE, 2021c, pp. 25, 34; similar in Reid, 1996, p. 4). If based on ""a model of all decision points[] and their outcomes"", then its coverage is ""calculated as the percentage of decision point outcomes executed by the test suite"" (Reid, 1996, p. 4). See Myers (1979)"
Defect-based Testing,"Technique (Hamburg and Mogyorodi, 2024; Souza et al., 2017, p. 3)","Testing ""in which test cases are developed from what is known about a specific defect type"" (Hamburg and Mogyorodi, 2024)","Dynamic Testing, Specification-based Testing, Behavioural Testing (Patton, 2006, pp. 87-88)",,"Related to the idea of a ""defect taxonomy"" (Hamburg and Mogyorodi, 2024). Patton (2006, p. 88) says if a specific defect is found, it is wise to look for other defects in the same location and for similar defects in other locations, but doesn't give this approach a name; this can be referred to as ""defect-based testing"" based on the principle of ""defect clustering"" (ChatGPT, 2024; similar in Rus et al., 2008). May be related to metamorphic testing"
Denial of Service,Approach,"""Security attack[s] … intended to overload the system with requests such that legitimate requests cannot be serviced"" (Hamburg and Mogyorodi, 2024)","Security Attacks (Hamburg and Mogyorodi, 2024), Stress Testing?","DoS (Hamburg and Mogyorodi, 2024)",
Design-based Testing,Technique?,"Testing ""in which test cases are designed based on the architecture and/or detailed design of a component or system"" (Kam, 2008, p. 44)",,,
Design-driven Testing,Approach,"Testing that ""verifies"" the ""system conforms to [the specified] design"" (Firesmith, 2015, p. 33)",,,
Desk Checking,"Technique (IEEE, 2017, p. 133)","A ""manual simulation of program execution"" or visual examination of code listings, test results, or other documentation to ""detect faults"" and "" identify errors"" (IEEE, 2017, p. 133; OG ISO/IEC 2015)",,,"Related to inspections and walkthroughs (IEEE, 2017, p. 133)"
Desktop Development Testing,"Level (implied by Gerrard, 2000a, p. 13)","Testing of web-based applications that focuses on, ""broadly, what the browser executes"" (Gerrard, 2000a, p. 13)","Web Application Testing, Development Testing?",,
"Deterministic Testing (implied by Washizaki, 2024, p. 5-16)","Technique (implied by Washizaki, 2024, p. 5-16)",,,,"In contrast to random testing; it is unclear if this is an actual test approach or simply a method for selecting input data, etc. for tests (Washizaki, 2024, p. 5-16)"
"Dev/Sec/Ops Testing? (inferred from Dev/Sec/Ops (Washizaki, 2024, p. 9-5) and DevOps testing (Firesmith, 2015, p. 29))",Practice?,"DevOps testing that ""applies security at all phases of the software life cycle"", including in the testing phase (Washizaki, 2024, p. 9-5)","DevOps Testing, Security Testing, Agile Testing, Shift-Left Testing (Washizaki, 2024, p. 9-5), Continuous Testing, Automated Testing (Washizaki, 2024, p. 9-5; implied by DevOps testing)",,
Developer Testing,Practice?,"Testing tied to ""developer code check-in/check-out procedures"" (Gerrard, 2000a, p. 11)","Role-based Testing (Firesmith, 2015, p. 39), Development Testing (usually) (IEEE, 2017, p. 136)",,See Extreme Programming Explained by Kent Beck
"Development Environment Testing (Firesmith, 2015, p. 25)",Technique?,,"Development Testing? (Gerrard, 2000a, p. 11)",,
"Development Organization Testing (Firesmith, 2015, p. 37)",Practice?,,"Organization-based Testing (Firesmith, 2015, p. 37), Development Testing",,
Development Testing,Level,"""Testing conducted during the development of a system or component, usually in the development environment by the developer … to establish whether … [it] satisfies its criteria"" (IEEE, 2017, p. 136)","Formal Testing, Informal Testing, System Testing (can be), Unit Testing (can be) (IEEE, 2017, p. 136)","Developmental Testing (DT)? (Firesmith, 2015, p. 30)","The secondary definition, ""testing conducted to establish whether a new software product or software-based system (or components of it) satisfies its criteria"" (IEEE, 2017, p. 136), seems to overlap with acceptance testing. Can be used to generate test cases for regression testing (Gerrard, 2000a, p. 11); is this sufficient for a ""parent"" relation? There are ""[m]any [t]ypes"" (Firesmith, 2015, p. 30), but no examples are given"
"Development Tool Testing (Firesmith, 2015, p. 25)",Technique?,,Development Testing?,,
Device-based Testing,Practice?,"""Testing in which test suites are executed on physical or virtual devices"" (Hamburg and Mogyorodi, 2024)",,,
"DevOps Testing (Firesmith, 2015, p. 29)",Practice?,"Testing performed in the context of DevOps: a ""set of principles and practices … for … specifying, developing, and operating software and systems products and services, and continuous improvements in all aspects of the life cycle"" (Washizaki, 2024, p. 10-7; OG [11])","Continuous Testing (Firesmith, 2015, p. 29; implied by Washizaki, 2024, pp. 10-7 (OG [11]), 11-11), Automated Testing (implied by Washizaki, 2024, p. 11-11)",,
"Differential Assertion Checking (DAC) (Lahiri et al., 2013)",Practice?,"The ""checking [of] two versions of a program with respect to a set of assertions"" to see if there is ""an environment in which [one version] passes but [another version] fails"" (Lahiri et al., 2013, p. 345)","Assertion Checking, Regression Testing (Lahiri et al., 2013, p. 346), Relative Correctness Testing, Back-to-Back Testing? (p. 345)",,
Disaster/Recovery Testing,"Type (IEEE, 2022, p. 22; 2021b, pp. 41, 43; 2021c, p. 37, Tab. A.1)","Testing to determine the degree to which a system can ""return to normal operation after a hardware or software failure"" (IEEE, 2017, p. 140) or if ""operation of the test item can be transferred to a different operating site and … be transferred back again once the failure has been resolved"" (2021c, p. 37)","Fault Tolerance Testing, Recoverability Testing (IEEE, 2021c, Tab. A.1), Model-based Testing, Requirements-based Testing, Conformance Testing, System Testing (can be) (p. 37), Risk-based Testing (can be) (Bas, 2024, p. 26), Maturity Testing (if it exists) (IEEE, 2021c, Tab. A.1), Non-functional Testing (implied by 2021b, p. 43)","Sometimes spelled without a slash (IEEE, 2021c, p. 37; implied by Tab. A.1; 2017, p. 140; 2013, p. 20; Washizaki, 2025, p. 6-9), DR Testing (implied by Bas, 2024, p. 26)","""Requires stopping the service, identifying the checkpoint state and triggering the failover process""; ""should be constantly rehearsed as changes to the production environment are made"" (Washizaki, 2024, p. 6-8) and may be done in a pre-production environment (IEEE, 2021b, p. 42). Difference between this and recovery testing (in the context of performance)?"
"Distributed Testing (Firesmith, 2015, p. 42; (in the context of backup testing) Bas, 2024, p. 16)",Approach,"Testing the ability of a system to process and ""store data on multiple storage nodes or servers working together in different places""? (Bas, 2024, p. 16)","Availability Testing, Reliability Testing, Load Balancing Testing, Efficiency Testing, Scalability Testing, Data Integrity Testing, Recovery Testing, Fault Tolerance Testing (implied by Bas, 2024, p. 16)",,
DOM Testing,"Technique (Bajammal and Mesbah, 2018, p. 199)","The ""use [of] a browser automation tool… to make assertions on the DOM … of the webpage"" (Bajammal and Mesbah, 2018, p. 193), such as the presence of certain elements and their properties and hierarchy (p. 199)","Web Application Testing (Bajammal and Mesbah, 2018, p. 193), Model-based Testing (implied by Doğan et al., 2014, p. 179)","Document Object Model Testing (Bajammal and Mesbah, 2018, p. 193)","Cannot be used for canvas elements (Bajammal and Mesbah, 2018, p. 193), although their visual information may be able to be ""represented as an augmented DOM inside the canvas element"" (Bajammal and Mesbah, 2018), which may be used to support test-driven development (p. 199)"
Domain Analysis,Approach,"""A combination of equivalence partitioning and boundary value analysis"" (IEEE, 2021c, p. 11; OG Beizer 1995)","Domain Testing?, Domain-based Testing?",,
"Domain Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Specification-based Testing, Structure-based Testing, Dynamic Testing (Peters and Pedrycz, 2000, Tab. 12.1), Infection-oriented Testing (Fig. 12.3)",,
"Domain-based Testing (Firesmith, 2015, p. 26)",Technique?,,,,
"Domain-independent Testing (Firesmith, 2015, p. 26)",Technique?,,"Domain-based Testing (Firesmith, 2015, p. 26)",,"There are ""[m]any [t]ypes"" (Firesmith, 2015, p. 26), but no examples are given"
"Domain-specific Testing (Firesmith, 2015, p. 26)",Technique?,,"Domain-based Testing (Firesmith, 2015, p. 26)",,"All examples given by Firesmith (2015, p. 26) are focused on hardware, but this might not be representative"
"DT Organization Testing (Firesmith, 2015, p. 37)",Practice?,,"Independent Test Organization Testing (Firesmith, 2015, p. 37), Development Testing? (Firesmith, 2015, p. 30)",,
Dynamic Analysis,Approach,"The ""process of evaluating a system or component based on its behavior during execution"" (IEEE, 2017, p. 149) ""with test data … to detect errors"" (Peters and Pedrycz, 2000, p. 438)","Dynamic Testing (inferred from its static counterpart (IEEE, 2022, pp. 9, 17, 25, 28; 2021a, pp. 3, 21; Hamburg and Mogyorodi, 2024; Gerrard, 2000a, Fig. 4, p. 12; 2000b, p. 3))",,"""Relies on the use of probes"" (Peters and Pedrycz, 2000, p. 438)"
Dynamic Testing,Approach,"""Testing in which a test item is evaluated by executing it"" (IEEE, 2022, p. 4; 2021a, p. 2)","V-Model Testing, W-Model Testing (Gerrard, 2000a, p. 9)","Software Testing (Peters and Pedrycz, 2000, p. 438; see \Cref{static-test}), Dynamic Analysis (p. 438; implied by IEEE, 2017, p. 149; Hamburg and Mogyorodi, 2024)","Can ""only occur in the parts of the life cycle when executable code is available"" (IEEE, 2022, p. 18)"
Each Choice Testing,"Technique (IEEE, 2022, p. 22; 2021c, pp. 2, 17, Fig. 2)","Testing that covers enough P-V pairs so that ""each parameter value is included at least once"" (IEEE, 2021c, p. 17)","Combinatorial Testing (IEEE, 2022, p. 22; 2021c, pp. 2, 17, Fig. 2), t-wise Testing","1-wise Testing (IEEE, 2021c, p. 17)","""The minimum number of test cases required to achieve 100% … [coverage is] the maximum number of values any one of the test item parameters can take"" (IEEE, 2021c, p. 17). See Grindal et al., 2005"
"Efficiency Testing (Kam, 2008, p. 44)","Type (implied by its quality (IEEE, 2017, p. 154; Hamburg and Mogyorodi, 2024))","Testing that evaluates the ""degree to which a system or component performs its designated functions with minimum consumption of resources"" (IEEE, 2017, p. 154) or ""the degree to which resources are expended in relation to results achieved"" (Hamburg and Mogyorodi, 2024)","Backup Testing (can be) (Bas, 2024, Tab. 3)",,"Has the following subcategories: ""execution"" and ""storage"" (IEEE, 2017, p. 154). Related to effectiveness testing (if it exists) (Hamburg and Mogyorodi, 2024). See the IREB Glossary"
Elasticity Testing,"Type (implied by its quality (Bertolino et al., 2019, p. 93:30))","Testing that evaluates the ability of a system to ""dynamically scal[e] up and down … resources as needed"" (Bertolino et al., 2019, p. 93:18; similar on p. 93:13) ""without compromising the capacity to meet peak utilization"" (Washizaki, 2025, p. 5-9)","Memory Management Testing, Resource Utilization Testing (Washizaki, 2025, p. 5-9), Non-functional Testing (p. 5-9; Bertolino et al., 2019, pp. 93:11, 93:19), Performance Testing (pp. 93:2, 93:20, 93:30), Cloud Testing (can be) (Washizaki, 2025, p. 5-27, Bertolino et al., 2019, pp. 93:8, 93:20, 93:26), Stress Testing",,"Some objectives are ""to control behaviors, to identify the resources to be (un)allocated, to coordinate events in parallel, and … to target scalability"" (Bertolino et al., 2019, p. 93:14; OG [8, 9, 17, 30, 40, 91]); these resources may include cloud, (p. 93:26) ""compute, memory, and storage resources"" (Washizaki, 2025, p. 5-9). ""Closely related"" to scalability testing and performance testing (Bertolino et al., 2019, p. 93:28). ""Further research … is needed to test elastic systems properly"" (p. 93:21)"
Elementary Comparison Testing,"Technique (Kam, 2008, p. 44)","Testing ""in which test cases are designed to execute combinations of inputs using the concept of condition determination coverage"" (Kam, 2008, p. 44)","Specification-based Testing (Kam, 2008, p. 44), Comparison Testing?",,
"Embedded Tester Testing (Firesmith, 2015, p. 39)",Practice?,,"Tester Testing (Firesmith, 2015, p. 39)",,
"Encryption Testing (Firesmith, 2015, p. 57)","Type (implied by Firesmith, 2015, p. 57)",,"Security Testing (Firesmith, 2015, p. 57; Bas, 2024, p. 28)",,
End-to-end Testing,"Type (Hamburg and Mogyorodi, 2024), Technique (Firesmith, 2015, p. 47; Sharma et al., 2021, pp. 601, 603, 605-606)","Testing ""in which business processes are tested from start to finish under production-like circumstances"" (Hamburg and Mogyorodi, 2024)","Specification-based Testing (Firesmith, 2015, p. 47), Systems Testing (Black, 2009, p. 14), System Integration Testing (p. 178), Integration Testing (Sharma et al., 2021, p. 603; Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Smoke Testing, Dynamic Testing, Post-Deployment Monitoring (2000a, Tab. 2; 2000b, Tab. 1), User Acceptance Testing (usually) (p. 31), Scenario-based Testing (implied by p. 31)","Sometimes spelled without hyphens (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1, p. 31; Sharma et al., 2021, pp. 601, 603, 605-606), E2E Testing (Hamburg and Mogyorodi, 2024)","Different from large scale integration testing (Gerrard, 2000b, p. 31)"
Endurance Testing,"Type (IEEE, 2013, p. 2; implied by Firesmith, 2015, p. 55), Technique (IEEE, 2021c, p. 38-39)","""Testing conducted to evaluate whether a test item can sustain a required [""significant"" (Hamburg and Mogyorodi, 2024)] load continuously for a specified period of time"" (IEEE, 2013, p. 2; similar in 2021c, p. 39; Hamburg and Mogyorodi, 2024) ""within the system's operational context"" (Hamburg and Mogyorodi, 2024)","Performance-related Testing, Model-based Testing, Requirements-based Testing, Conformance Testing (IEEE, 2021c, p. 38), Performance Efficiency Testing (2013, p. 2), Reliability Testing (Firesmith, 2015, p. 55)","Soak Testing (IEEE, 2021c, p. 39)",
Entry Point Testing,"Technique (implied by Reid, 1996, p. 4)","Testing that ""exercise[s] all entry points"" (Reid, 1996, p. 4)","Control Flow Testing, MC/DC Testing (Reid, 1996, p. 4), Branch Testing (can be) (IEEE, 2021c, pp. 23-24), Decision Testing (often but incorrectly) (Reid, 1996, p. 4)",,
"Equivalence Checking (Lahiri et al., 2013, p. 345)",Technique?,"Verifying the ""semantic equivalence"" between ""previous versions of an evolving program …  to ensure the correctness of [a] transformation"", such as refactoring (Lahiri et al., 2013, p. 345; OG [25], [12], [19])","Incremental Testing? (by ""carrying over invariants that are unaffected by the syntactic changes"" (Lahiri et al., 2013, p. 345; OG [28]) if the original does ""not have any false warnings"" (p. 345)), Relative Correctness Testing, Regression Testing? (p. 354)",,"""Applicable in very limited contexts … since most software changes … induce some behavioral change"" (Lahiri et al., 2013, p. 345)"
Equivalence Partitioning,"Technique (IEEE, 2022, pp. 4, 20, 22; 2021a, p. 5; 2021c, pp. 4, 8, 10, Fig. 2; 2013, p. 3; Washizaki, 2024, p. 5-11; Firesmith, 2015, p. 48; Souza et al., 2017, p. 3; Reid, 1996, pp. 2, 7)","Testing ""designed to exercise equivalence partitions by using one or more [arbitrary (IEEE, 2021c, p. 11)] representative members of each partition"" (2022, p. 4; 2021c, p. 4; 2013, p. 3; similar in Patton, 2006, p. 67) ""based on a specified criterion or relation"" to create ""a representative test suite"" (Washizaki, 2024, p. 5-11)","Specification-based Testing (IEEE, 2022, p. 22; 2021c, pp. 4, 10, Fig. 2; Washizaki, 2024, p. 5-11; Hamburg and Mogyorodi, 2024; Sharma et al., 2021, Fig. 1; Peters and Pedrycz, 2000, Fig. 12.2; Souza et al., 2017, p. 3; Reid, 1996, p. 7), Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, pp. 3, 20; 2021c, pp. 6, 10), Positive Testing, Negative Testing (IEEE, 2021c, pp. 10-11; Reid, 1996, p. 7), Domain Analysis (IEEE, 2021c, p. 11; OG Beizer 1995), Unit Testing (can be) (IEEE, 2021b, p. 55; Reid, 1996, pp. 1-2, 6-7), One-to-One Testing, Minimized Testing (can be) (IEEE, 2021c, p. 11; OG Myers 1979), Usability Testing, Classification Tree Method? (can be) (p. 10), Grey-Box Testing (Firesmith, 2015, p. 48), Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5), Functional Testing? (IEEE, 2022, p. 20), Sampling (implied by IEEE, 2022, p. 18; Washizaki, 2024, p. 5-2),  Boundary Value Analysis (often) (Reid, 1996, p. 7)","EP (Reid, 1996, pp. 1, 7), Partition Testing (Hamburg and Mogyorodi, 2024; OG Beizer), Equivalence Class Testing (Firesmith, 2015, p. 48), Equivalence Classing (Patton, 2006, p. 67; implied by IEEE, 2022, p. 4; 2021c, p. 4), Equivalence Partitioning Testing (implied by Kam, 2008, p. 7), Equivalence Testing? (Sharma et al., 2021, Fig. 1)","Default, empty, blank, null, zero, and none values should be their own equivalence class, since ""the software usually handles them differently"" than ""the valid cases or … invalid cases"" (Patton, 2006, p. 78)"
Ergonomics Testing,Approach,"""Testing to determine whether a component or system and its input devices are being used properly with correct posture"" (Hamburg and Mogyorodi, 2024)",,,Does this test the system or its use?
Error Forcing,Technique?,"Setting variables to specific values to see how errors are handled (Patton, 2006, p. 116)","Data Testing (Patton, 2006, p. 116), Error Tolerance Testing (implied by p. 116)",,"Any error forced must have a chance of occurring in the real world, even if it is unlikely, and must be double-checked for validity (Patton, 2006, p. 116)"
Error Guessing,"Technique (IEEE, 2022, pp. 4, 34, Fig. 2; 2021c, pp. iii-iv, 4, 11, 29, 35, 122, 125, Fig. 2, Tab. A.2; 2013, pp. 3, 33; Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 50), Practice (IEEE, 2013, p. 33)","Testing based on a ""checklist of potential defects [that] can be derived by various means, such as taxonomies of known errors, information contained in incident management systems, from a tester's knowledge, experience or understanding of the test item(s) or similar test items or from the knowledge of other stakeholders (e.g. system users or programmers)"" (IEEE, 2021c, p. 29; similar in Patton, 2006, pp. 88-89), ""to anticipate the most plausible faults in each SUT"" (Washizaki, 2025, p. 5-13)","Experience-based Testing (IEEE, 2022, pp. 4, 22; 2021c, pp. 4, 11, 29, Fig. 2; Washizaki, 2025, p. 5-13; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 50; Sharma et al., 2021, Fig. 1; implied by Reid, 1996, p. 7), Model-based Testing, Checklist-based Testing (IEEE, 2021c, p. 29), Fault-based Testing (Bourque and Fairley, 2014, p. 4-9), Error-based Testing (implied by van Vliet, 2000, p. 399)","Error Guessing Testing (Firesmith, 2015, p. 50)","""There is currently no industry agreed approach to calculating coverage"" (IEEE, 2021c, p. 35). See Myers (1979)"
Error Tolerance Testing,"Type (implied by its quality (IEEE, 2017, p. 166); Firesmith, 2015, p. 56)","Testing the ""ability of a system or component to continue normal operation despite the presence of erroneous inputs"" (IEEE, 2017, p. 166) ""that should be rejected"" (Kam, 2008, p. 45)","Robustness Testing (Firesmith, 2015, p. 56)","Invalid Testing (implied by Kam, 2008, p. 45)","Related to fault tolerance testing and robustness testing (IEEE, 2017, p. 166)"
Error-based Testing,"Technique (van Vliet, 2000, p. 399)","Testing that ""focus[es] on error-prone points, based on knowledge of the typical errors that people make"" (van Vliet, 2000, p. 399)","Error-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.4), Experience-based Testing",,
"Error-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.4)",Approach,"Testing ""motivated by the potential presence of errors in the programming process"" (Peters and Pedrycz, 2000, p. 440)",,,
Event Space Testing,Technique (inferred from coverage-based testing),"Testing that aims to maximize the ""number of events in the GUI space of the SUT exercised by [the] test suite"" (Doğan et al., 2014, Tab. 13; OG Saxena et al., 2010)","Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13), GUI Testing, Control Flow Testing?","All-Events Testing? (Doğan et al., 2014, Tab. 13; OG Mansour and Houri, 2006)",
Evidence-based Testing,"Technique (Washizaki, 2024, p. 5-12)","Testing that ""follows a rigorous research approach"", including ""identifying"" and ""critically analyzing the evidence in light of the problem"" (Washizaki, 2024, p. 5-12)","Specification-based Testing (Washizaki, 2024, p. 5-12)",,"""Evidence-based software engineering (EBSE) … is the _best_ solution for a practical problem"" (Washizaki, 2024, p. 5-12); related to experience-based testing?"
Exhaustive Testing,"Approach (IEEE, 2022, p. 4; Hamburg and Mogyorodi, 2024)","Testing ""in which all combinations of input values and preconditions are tested"" (IEEE, 2022, p. 4; similar in Hamburg and Mogyorodi, 2024)","Dynamic Testing (IEEE, 2022, p. 18)","Complete Testing (Hamburg and Mogyorodi, 2024; Kam, 2008, p. 6), Branch Condition Combination Testing (if ""each subcondition is viewed as a single input"") (Peters and Pedrycz, 2000, p. 464)","Impossible in most non-trivial situations (IEEE, 2022, p. 4; Washizaki, 2024, p. 5-5; Peters and Pedrycz, 2000, pp. 439, 461; van Vliet, 2000, p. 421) since it is ""prohibitively expensive"" (Mandl, 1985, p. 1058). Can be defined many different ways (Kaner et al., 2011, p. 7)"
"Exit Point Testing (inferred from entry point testing; Reid, 1996, p. 4)","Technique (implied by Reid, 1996, p. 4)","Testing that exercises all exit points (inferred from entry point testing; Reid, 1996, p. 4)","Control Flow Testing, MC/DC Testing (Reid, 1996, p. 4), Branch Testing (can be) (IEEE, 2021c, p. 24), Decision Testing (often but incorrectly) (Reid, 1996, p. 4)",,
Experience-based Testing,"Practice (IEEE, 2022, Fig. 2; 2021c, p. viii; 2013, pp. iii, 31, 33), Technique (IEEE, 2022, Fig. 2; Firesmith, 2015, pp. 46, 50)","Testing based on testers' experience (IEEE, 2022, p. 4; 2021c, p. 4; Washizaki, 2025, p. 5-13; Hamburg and Mogyorodi, 2024), knowledge, and intuition (Washizaki, 2025, p. 5-13; Hamburg and Mogyorodi, 2024), as well as the SUT's context (Washizaki, 2025, p. 5-13)","Dynamic Testing (IEEE, 2022, p. 17; Patton, 2006, pp. 88-89; Sharma et al., 2021, Fig. 1), Specification-based Testing (Patton, 2006, pp. 89-89), Unscripted Testing (often) (IEEE, 2022, p. 34)","Sometimes spelled without a hyphen (Sharma et al., 2021, Fig. 1)","Can ""target … quality areas"" (IEEE, 2022, p. 4; 2021c, p. 4)"
Expert Usability Reviews,Approach,"""Informal usability review[s] in which the reviewers are … usability experts [and/]or subject matter experts"" (Hamburg and Mogyorodi, 2024)","Informal Testing, Usability Testing, Reviews (Hamburg and Mogyorodi, 2024)",,
Exploratory Testing,"Practice (IEEE, 2022, pp. 11, 20, 34, Fig. 2; 2021a, p. 5; 2021c, p. viii; 2013, pp. 13, 33; implied by 2022, p. 33), Technique (Washizaki, 2025, pp. 5-13 to 5-14; Firesmith, 2015, p. 50)","""Simultaneous learning, test design and test execution"" (Washizaki, 2025, p. 5-14; similar in IEEE, 2021a, p. 2; Hamburg and Mogyorodi, 2024) that is ""spontaneous"" and aims to find ""hidden properties"" that ""can interfere with other properties of the software under test"" (IEEE, 2022, p. 5; 2013, p. 3)","Experience-based Testing (IEEE, 2022, p. 5; 2021a, p. 2; 2021c, p. viii; 2013, p. 3; Washizaki, 2025, p. 5-14; Sharma et al., 2021, Fig. 1; Firesmith, 2015, p. 50), Unscripted Testing (IEEE, 2022, p. 33; 2021a, p. 2; 2017, p. 174; Firesmith, 2015, p. 45; implied by Washizaki, 2025, p. 5-14), Specification-based Testing (Patton, 2006, p. 65; often in IEEE, 2022, p. 33), Dynamic Testing, Behavioural Testing (Patton, 2006, p. 65), Informal Testing (Kam, 2008, p. 44; usually in IEEE, 2022, p. 33), Defect-based Testing (often), Requirements-based Testing (often), Coverage-based Testing (often) (IEEE, 2022, p. 33)","Unscripted Testing (implied by Kuļešovs et al., 2013, p. 214)","""Test cases are not defined in advance but are dynamically designed, executed, and modified according to the collected evidence"" (Washizaki, 2025, p. 5-14), but this process is often structured with ""session sheets"", which are also ""used to capture information about what was tested, and any anomalous behaviour observed"" (2022, p. 33), or ""test charters"" (Hamburg and Mogyorodi, 2024), but the actual results might not be recorded when they ""match the expected results"" (IEEE, 2021a, p. 40). Patton says this is used when a specification is not available to determine and test the software's features, and finding any bugs through this process is a good thing (2006, p. 65). ""Widely used in shift-left development (such as Agile)"" (Washizaki, 2025, p. 5-14). See Whittaker (2010)"
"Expression Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Infection-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",,
Extended Entry Table Testing,"Technique (inferred from decision table testing and IEEE, 2021c, p. 18)","Testing ""based on exercising decision rules"" (IEEE, 2022, p. 4) in an extended entry table (2021c, p. 18; 2017, p. 175; OG ISO, 1984)","Decision Table Testing (IEEE, 2021c, p. 18; 2017, p. 175; OG ISO, 1984), Equivalence Partitioning (IEEE, 2021c, p. 18)",,
"External Links Integration (Testing) (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1)",Level (inferred from integration testing),,"Functional Testing, Dynamic Testing, Large Scale Integration Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Integration Testing, Web Application Testing",,
"Extreme Value Testing (Kam, 2008, p. 7)",Technique (inferred from boundary value analysis),,"Specification-based Testing (Kam, 2008, p. 7), Boundary Value Analysis",,
"E-business Testing (Gerrard, 2000b, p. 30)",Approach,,,,
Factory Acceptance Testing,"Level (IEEE, 2022, p. 22; Firesmith, 2015, p. 30; inferred from acceptance testing)",,"Acceptance Testing (IEEE, 2022, p. 22; Firesmith, 2015, p. 30)","FAT (Firesmith, 2015, p. 30)",
Failover Testing,Approach,"Testing that ""validates the SUT's ability to manage heavy loads or unexpected failure to continue typical operations"" (Washizaki, 2025, p. 5-9)","Non-functional Testing (Washizaki, 2025, p. 5-9), Continuity Testing (Koomen et al., 2006, p. 50), Bottom-Up Testing (Gerrard, 2000b, p. 27), Failure Tolerance Testing? (Firesmith, 2015, p. 56), Data Center Testing? (p. 24)","Failover and Recovery Testing (implied by Firesmith, 2015, p. 56), Failover and Restore Testing (implied by p. 24)","Failover can be achieved ""by allocating extra resources"" (Washizaki, 2025, p. 5-9) or entering a ""backup operational mode in which [these responsibilities] … are assumed by a secondary system"" (Hamburg and Mogyorodi, 2024). This testing ""should be constantly rehearsed as changes to the production environment are made"" (Washizaki, 2025, p. 6-9). Failures are simulated ""with an automated load running to explore system behaviour in production situations"" (Gerrard, 2000b, p. 27). Related to recoverability validation (Washizaki, 2025, p. 5-9)"
Failover/Recovery Testing,"Type (IEEE, 2021c, p. 37)","Testing that determines the ability ""to mov[e] to a back-up system in the event of failure, without transfer[ing] to a different operating site"" (IEEE, 2021c, p. 37)","Disaster/Recovery Testing (IEEE, 2021c, p. 37), Continuity Testing (implied by Koomen et al., 2006, p. 50)","Failover/Restore Testing? (implied by Koomen et al., 2006, p. 50)",
"Failure Tolerance Testing (Firesmith, 2015, p. 56)","Type (implied by Firesmith, 2015, p. 56)",,"Robustness Testing (Firesmith, 2015, p. 56)",,
Fault Injection Testing,Approach,"Testing where ""faults are artificially introduced into the SUT"" (Washizaki, 2024, p. 5-18) to, at least partially, ""test the robustness of the system in the event of internal and external failures"" (IEEE, 2022, p. 42) and ""whether it can detect and possibly recover from [them]"" (Hamburg and Mogyorodi, 2024)","Robustness Testing (IEEE, 2022, p. 42; Ghosh and Voas, 1999, p. 40), Fault Tolerance Testing (Firesmith, 2015, p. 56), Security Testing (Ghosh and Voas, 1999, p. 40), Interface Testing (can be) (p. 42), Stress Testing (can be) (inferred from p. 42)","Fault Injection, Fault Injection Analysis (Ghosh and Voas, 1999, p. 39)",
Fault Seeding,Practice?,"The ""process of intentionally adding known faults to those already in a computer program … [to] estimat[e] the number of faults remaining"" (IEEE, 2017, p. 165) based on the ratio between the number of new faults and the number of introduced faults that were discovered (van Vliet, 2000, p. 427)",,"Error Seeding (IEEE, 2017, p. 165; van Vliet, 2000, p. 427; implied by Firesmith, 2015, p. 34), Bug Seeding (IEEE, 2017, p. 165), Bebugging (Hamburg and Mogyorodi, 2024)","Makes many assumptions, including ""that both real and seeded faults have the same distribution"", and requires careful consideration as to which faults are introduced and how (van Vliet, 2000, p. 427)"
"Fault Sensitivity Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Infection-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",,
Fault Tolerance Testing,"Type (implied by its quality (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1; 2017, pp. 38, 375; Washizaki, 2024, p. 7-10); Firesmith, 2015, p. 56)","Testing the ""capability of a product to operate as intended despite the presence of hardware or software faults"" (ISO/IEC, 2023a; similar in Washizaki, 2024, p. 7-10) ""by detecting errors and then recovering from them or containing their effects if recovery is not possible"" (Washizaki, 2024, p. 4-11)","Reliability Testing (IEEE, 2021c, Tab. A.1; 2017, p. 375; Washizaki, 2024, p. 7-10), Availability Testing (IEEE, 2017, p. 38), Robustness Testing (Firesmith, 2015, p. 56), Dependability Testing (if it exists) (ISO/IEC, 2023a; IEEE, 2017, p. 375)","Sometimes spelled with a hyphen (Ghosh and Voas, 1999, p. 44), Robustness Testing (Hamburg and Mogyorodi, 2024)","May be supported by fault injection (Ghosh and Voas, 1999, pp. 40, 44)"
Fault Tree Analysis,"Technique (implied by Hamburg and Mogyorodi, 2024)","""A technique for analyzing the causes of failures that uses a hierarchical model of events and their logical relationships"" (Hamburg and Mogyorodi, 2024)",Static Analysis,,"May use ""fault"" and ""failure"" more generically as opposed to using their software-specific definitions"
Fault-based Testing,"Technique (Washizaki, 2024, p. 5-14; van Vliet, 2000, p. 399; implied by Barbosa et al., 2006, p. 3)","Testing that ""devise[s] test cases specifically to reveal likely or predefined fault categories"" (Washizaki, 2024, p. 5-14; similar in van Vliet, 2000, p. 399)","Error-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.4)",,"May be supported by fault injection testing (van Vliet, 2000, p. 399). See (Doğan et al., 2014, Tab. 27) for a collection of fault models/bug taxonomies for web application testing"
Feature Interaction Testing,"Level (Black, 2009, p. 11)",,Feature-based Testing,,
"Feature-based Testing (Firesmith, 2015, p. 28)",Approach,,"Unit Testing? (implied by Firesmith, 2015, p. 28)",,
"Features Testing (Gerrard, 2000a, Fig. 5)",Approach,,"Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5), Integration Testing? (implied by Gerrard, 2000a, Fig. 5; Sangwan and LaPlante, 2006, p. 26), System Testing? (Sangwan and LaPlante, 2006, p. 26)",,
Field Testing,Approach,"""Testing conducted to evaluate the system behavior under productive connectivity conditions in the field"" (Hamburg and Mogyorodi, 2024)","Beta Testing? (implied by Kam, 2008, p. 44), Operational (Acceptance) Testing?",Operational (Acceptance) Testing?,
"Flexibility Testing (Firesmith, 2015, p. 53)","Type (implied by its quality (ISO/IEC, 2023a; IEEE, 2017, p. 184); Firesmith, 2015, p. 53)","Testing the ""capability of a product to be adapted to changes in its requirements, contexts of use, or system environment"" (ISO/IEC, 2023a; similar in IEEE, 2017, p. 184)",,"Portability Testing (ISO/IEC, 2023a)",
Follow-on Operational Testing,Level (inferred from operational testing),,"Operational Testing (Firesmith, 2015, p. 30)","FOT (Firesmith, 2015, p. 30)",
Forcing Exception Testing,"Technique (Washizaki, 2024, p. 5-12)","Testing that ""check[s] whether the SUT can manage a predefined set of exceptions/errors"" (Washizaki, 2024, p. 5-12)","Negative Testing (usually), Scenario-based Testing? (Washizaki, 2025, p. 5-12)","Testing-to-Fail (implied by Patton, 2006, pp. 66-67, 78), Error Forcing (implied by p. 67)",
"Formal Methods (Kam, 2008, Tab. 1)",Approach,,Formal Testing?,,
"Formal Modular Verification (Chalin et al., 2006, p. 342)",Approach,,Formal Testing,,"Does this imply ""modular verification""?"
Formal Testing,Technique (inferred from informal testing),"""Testing conducted in accordance with test plans and procedures that have been reviewed and approved"" (IEEE, 2017, p. 188) with ""formally documented output[s]"" (Hamburg and Mogyorodi, 2024)",,,"Since informal testing still follows ""test plans and procedures"" (IEEE, 2017, p. 220), this is probably not related to scripted testing (at least specifically). In the domain of web testing, this can be split into TestUML and StateChart (Kam, 2008, pp. 4-5)"
Formative Evaluations,Approach,"""Evaluation[s] designed and used to improve the quality of a component or system, especially when it is still being designed"" (Hamburg and Mogyorodi, 2024)",Development Testing?,,
"Full Conformance Testing (inferred from full conformance (IEEE, 2021a, pp. 9-10; 2021b, pp. 5, 37; 2021c, p. 7) and conformance testing)",Type (inferred from conformance testing),"Testing that ""all of the requirements … of the chosen (non-empty) set of techniques … and the corresponding test coverage measurement approaches … have been satisfied"" (IEEE, 2021c, p. 7)","Conformance Testing, Requirements-based Testing",,
"Functional Configuration Audit (IEEE, 2015, p. 10)",Practice (inferred from audits),,"Audits, Configuration Testing, Functional Testing","FCAs (IEEE, 2015, p. 10)",
"Functional Suitability Testing (IEEE, 2022, p. 7; 2017, p. 337)","Type (IEEE, 2022, p. 7; 2017, p. 196; implied by its quality (ISO/IEC, 2023a; IEEE, 2021c, p. viii, Tab. A.1))","Testing to determine the ""capability of a product to provide functions that meet stated and implied needs of intended users when it is used under specified conditions"" (ISO/IEC, 2023a; similar in IEEE, 2017, p. 196; Hamburg and Mogyorodi, 2024)","Conversion Testing, Functional Testing, Localization Testing (IEEE, 2021c, Tab. A.1)","Functionality Testing (although this seems wrong) (implied by Hamburg and Mogyorodi, 2024)","Related to interoperability testing (Hamburg and Mogyorodi, 2024)"
Functional Testing,"Type (IEEE, 2022, pp. 15, 20, 22; 2021a, pp. 8, 11; 2021b, p. 41; 2021c, pp. 7, 38, Tab. A.1; 2017, p. 473; 2016, p. 4; Hamburg and Mogyorodi, 2024; Koomen et al., 2006, p. 50; implied by the quality of ""correctness"" (IEEE, 2017, p. 104; Washizaki, 2025, p. 3-13)), Technique (Barbosa et al., 2006, p. 3; inferred from specification-based testing)","Testing ""used to check the implementation of functional requirements"" (IEEE, 2022, p. 21; similar in 2021c, p. 38; 2017, p. 196; Washizaki, 2025, p. 5-7; Peters and Pedrycz, 2000, Tab. 12.2) by ""focus[ing] solely on the outputs generated in response to selected inputs and execution conditions"" (IEEE, 2017, p. 196; similar in Peters and Pedrycz, 2000, Tab. 12.2) ""to verify that the SUT conforms to standards, rules, specifications, requirements, design, processes, or practices"" (Washizaki, 2025, p. 5-7)","Dynamic Testing (IEEE, 2021a, p. 11), Requirements-based Testing, Structure-based Testing (2021c, p. 38), Specification-based Testing (p. 38; Kam, 2008, p. 42; implied by Washizaki, 2025, p. 5-7), Functional Suitability Testing (IEEE, 2021c, Tab. A.1), Automated Testing (can be) (Gerrard, 2000a, p. 11)","Specification-based Testing (IEEE, 2017, p. 196; van Vliet, 2000, p. 399; Kam, 2008, pp. 44-45, 48; implied by IEEE, 2021c, p. 129; 2017, p. 431), Conformance Testing (Washizaki, 2025, p. 5-7; implied by IEEE, 2017, p. 93), Correctness Testing (Washizaki, 2025, p. 5-7)","May be supported by symbolic analysis and/or execution (Peters and Pedrycz, 2000, Tab. 12.2)"
Functionality Testing,"Type (implied by its quality (IEEE, 2017, pp. 196, 419, 422); Firesmith, 2015, p. 53)","Testing of the ""capabilities of the various … features provided by a product"" (IEEE, 2017, p. 196)","Build Verification Testing (Hamburg and Mogyorodi, 2024), Regression Testing (can be) (ISO/IEC, 2014c), Smoke Testing?, Functional Suitability Testing?",,"Should be performed ""in all projects"", possibly implicitly ""during development and user evaluation"", but is ""typically the … most expensive and time consuming to implement and execute"" (Gerrard, 2000a, p. 13). Difference between this and functional testing? Gerrard makes a distinction between them (2000a, Tab. 2; 2000b, Tab. 1)"
"Functions Testing (implied by Doğan et al., 2014, Tab. 11)",Technique (inferred from coverage-based testing),,"Coverage-based Testing (Doğan et al., 2014, Tab. 11), Structure-based Testing (implied by Tab. 11), Requirements-based Testing (Kam, 2008, p. 47), Control Flow Testing?",,Difference between this and method testing?
Fuzz Testing,"Technique (Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 51), Practice (inferred from mathematical-based testing)","Testing where ""high volumes of random (or near random) data, called fuzz, are used to generate [test] inputs"" (IEEE, 2022, p. 5; similar in Hamburg and Mogyorodi, 2024) ""aimed at breaking the software"" (Bourque and Fairley, 2014, p. 4-8) or ""discover[ing] security vulnerabilities"" (Hamburg and Mogyorodi, 2024)","Mathematical-based Testing (IEEE, 2022, p. 36), Security Testing (Hamburg and Mogyorodi, 2024; often in Bourque and Fairley, 2014, p. 4-8; can be in Godefroid and Luchaup, 2011, p. 23), Random Testing (Bourque and Fairley, 2014, p. 4-8; Firesmith, 2015, p. 51; implied by IEEE, 2022, p. 5; Hamburg and Mogyorodi, 2024)","Fuzzing (Hamburg and Mogyorodi, 2024)","This is tagged (?) as ""artificial intelligence"" (IEEE, 2022, p. 5), but I don't think AI is required"
Galumphing,"Technique (Firesmith, 2015, p. 50)",,"Exploratory Testing (Firesmith, 2015, p. 50)",,
Grey-Box Testing,"Technique (IEEE, 2021c, p. 8; Firesmith, 2015, pp. 46, 48; Sakamoto et al., 2013, p. 344)","Testing that ""utilis[es] a combination of knowledge from the test item's specification and structure"" (IEEE, 2021c, p. 8), such as ""the internal data structures and algorithms"" (Sakamoto et al., 2013, p. 344) with more of a focus on the specification (p. 344; Patton, 2006, p. 218)","Structure-based Testing (IEEE, 2021c, p. 8; Patton, 2006, p. 218; Sakamoto et al., 2013, p. 344), Specification-based Testing (IEEE, 2021c, p. 8; Patton, 2006, p. 218)","Sometimes spelled ""gray-box"" (Patton, 2006, p. 220; Sakamoto et al., 2013, p. 344) or ""graybox"" (Firesmith, 2015, pp. 46, 48)","Checking HTML tags is an example, since ""HTML doesn't execute or run, it just determines how text and graphics appear onscreen"" (Patton, 2006, p. 220)"
"Group Testing (Firesmith, 2015, p. 36)",Approach,,,,
GUI Testing,Approach,"""Testing performed by interacting with the software under test via the graphical user interface"" (Hamburg and Mogyorodi, 2024)","Browser Page Testing (implied by Gerrard, 2000b, p. 13), Dynamic Testing",,"See Banerjee et al., 2013; Sprenkle et al., 2012; Mariani et al., 2012; Andrews et al., 2005"
"Heartbeat (Firesmith, 2015, p. 31)",Practice?,,"Periodic Built-In Testing, Self-Testing (can be) (Firesmith, 2015, p. 31)",,Not sure what this is; investigate
Heuristic Evaluations,"Technique (Hamburg and Mogyorodi, 2024)","""Usability review[s] … that evaluate[] a work product by using a set of heuristics"" (Hamburg and Mogyorodi, 2024)","Usability Reviews (Hamburg and Mogyorodi, 2024), Usability Testing (Gerrard, 2000a, p. 13; 2000b, p. 22)",,"See [18] in Gerrard, 2000b, p. 22"
"High Frequency Testing (Sharma et al., 2021, pp. 601, 603, 605-606; OG [19])","Technique (Sharma et al., 2021, pp. 601, 603, 605-606)",,"Integration Testing (Sharma et al., 2021, p. 603)",,
High-Level Testing,Approach,"Testing ""with concrete values for preconditions, input data, expected results, postconditions, and a detailed description of actions (where applicable)"" (Hamburg and Mogyorodi, 2024)",,,"OG definition was about test cases, not a test approach"
"Human Factors Engineer Testing (Firesmith, 2015, p. 39)",Practice?,,"Developer Testing (Firesmith, 2015, p. 39)",,
"Human-in-the-Loop Testing (Firesmith, 2015, p. 23)",Level (inferred from system testing),,"System Testing (Firesmith, 2015, p. 23)","HIL Testing (Firesmith, 2015, p. 23)",
"Hyperlink Testing (Doğan et al., 2014, Tab. 13; OG Ricca and Tonella, 2001; OG 2002)",Technique (inferred from coverage-based testing),"Testing in which ""every hyperlink from every page in the site is traversed at least once"" (Doğan et al., 2014, Tab. 13; OG Ricca and Tonella, 2001; OG 2002)","Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13), Smoke Testing, Desktop Development Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Grey-Box Testing? (Patton, 2006, p. 220), Page Testing, Control Flow Testing?","All-Hyperlinks Testing? (Doğan et al., 2014, Tab. 13; OG Mansour and Houri, 2006), Link Checking?",
Hypothesis Testing,Approach,"""Validation of a theory and its assumptions using sample data"" (Hamburg and Mogyorodi, 2024)",,,"Ambiguous: what is a ""theory""? why use ""sample data"" and not just ""data""? is it specific to software or does it just refer to general science? Potentially related to model verification"
"Implementation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,"Testing ""guided by information derived from the implementation"" (Peters and Pedrycz, 2000, p. 440; OG Howden, 1975) where ""each execution of a program exercises a particular path"" (Peters and Pedrycz, 2000, p. 440)",,Structure-based Testing?,
"In-container Testing (Doğan et al., 2014, Tab. 8)",Approach,"""Testing in application servers"" (Doğan et al., 2014, Tab. 8)","Web Application Testing (Doğan et al., 2014, Tab. 8)",,
"Incremental Testing (Firesmith, 2015, p. 29)",Practice?,"Testing that ""occur[s] in an overlapping … manner"" to other phases of development (IEEE, 2017, p. 218) ""through a series of iterations"" (Washizaki, 2024, p. 10-5; OG [3, 8, 13]) or ""where components or systems are integrated and tested one or some at a time"" (Kam, 2008, p. 45)","Lifecycle-based Testing (Washizaki, 2024, p. 10-5; OG [2, 3, 10]; Firesmith, 2015, p. 29)","Incremental Verification? (implied by Lahiri et al., 2013, p. 345)",
"Independent Test Organization Testing (Firesmith, 2015, p. 37)",Practice?,,"Organization-based Testing (Firesmith, 2015, p. 37), Independent Tester Testing",,
"Independent Tester Testing (Firesmith, 2015, p. 39)",Practice?,,"Tester Testing (Firesmith, 2015, p. 39)",,
"Individual Testing (Firesmith, 2015, p. 36)",Approach,,,,
Inductive Assertion Methods,"Technique (IEEE, 2017, p. 220; OG ISO/IEC, 2002)","The process ""in which assertions are written describing program inputs, outputs, and intermediate conditions, a set of theorems is developed relating satisfaction of the input assertions to satisfaction of the output assertions, and the theorems are proved or disproved using proof by induction"" (IEEE, 2017, p. 220; OG ISO/IEC, 2002)","Proofs of Correctness (IEEE, 2017, p. 220; OG ISO/IEC, 2002), Mathematical-based Testing, Assertion Checking, Static Testing?",,
Industrial Web Application Testing,Approach,,"Web Application Testing (Kam, 2008, p. 10)",,
"Infection-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Implementation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",,
Informal Testing,"Technique (implied by Kam, 2008, p. 6)","""Testing conducted in accordance with test plans and procedures that have not been reviewed and approved by a customer, user, or designated level of management"" (IEEE, 2017, p. 220) without ""formally documented output"" (Hamburg and Mogyorodi, 2024)",,,"""Widely accepted by industry"" (which I think is wild) and ""the tedious and confusing software testing jargon"" is a drawback (Kam, 2008, p. 6) (although this isn't specific to informal testing)"
Infrastructure Testing,"Type (implied by Firesmith, 2015, p. 57), Level (implied by Gerrard, 2000a, p. 13; see \Cref{tab:ieeeCats})","Testing of ""infrastructure components[, or ""what runs on the servers"" (Gerrard, 2000a, p. 13),] to reduce the chances of downtime and improve the performance of the IT infrastructure"" (Washizaki, 2024, p. 5-9)","Security Testing (Firesmith, 2015, p. 57), Non-functional Testing (Washizaki, 2025, p. 5-9)",,
Initial Operational Testing,Level (inferred from operational testing),,"Operational Testing (Firesmith, 2015, p. 30)","IOT (Firesmith, 2015, p. 30)",
"Initial Testing (Firesmith, 2015, p. 34)",Approach,,,,
Input Data Testing,"Level (Hamburg and Mogyorodi, 2024)","Testing ""that focuses on the quality of the data used for training and prediction by ML models"" (Hamburg and Mogyorodi, 2024)",ML Model Testing,,
"Input Domain Testing (Peters and Pedrycz, 2000, Fig. 12.2)",Approach,,"Specification-based Testing (Peters and Pedrycz, 2000, Fig. 12.2)",,
"Input Validation Testing (Gerrard, 2000a, Fig. 5; Doğan et al., 2014, Tab. 13; OG Liu and Kuan Tan, 2008)",Technique (inferred from path testing),"Testing that ensures ""at least one path in the program's CFG w.r.t. the validation of inputs has been covered"" (Doğan et al., 2014, Tab. 13; OG Liu and Kuan Tan, 2008)","Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5), Path Testing, Data-driven Testing?",,"Related to the idea of ""input validation coverage (IVC)"" (Doğan et al., 2014, Tab. 13; OG Liu and Kuan Tan, 2008)"
"Input-Parameter Testing (Doğan et al., 2014, Tab. 13; OG Halfond and Orso, 2007)",Technique (inferred from combinatorial testing),"Testing that aims to ""cover[] all possibilities of input parameters"" (Doğan et al., 2014, Tab. 13; OG Halfond and Orso, 2007)","Web Application Testing, Specification-based Testing (Doğan et al., 2014, Tab. 13), Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, p. 3; 2021c, p. 6), Combinatorial Testing",,
Insourced Testing,Practice?,"""Testing performed by people who are co-located with the project team but are not fellow employees"" (Hamburg and Mogyorodi, 2024)",,,"Also mentioned by Firesmith (2015, p. 41)"
(Code) Inspections,"Technique (IEEE, 2017, p. 227), Level (Washizaki, 2025, p. 5-13)","""Visual examination[s] of … [code] to detect and identify software anomalies, including errors and deviations from standards and specifications"" (IEEE, 2017, p. 227; OG IEEE, 2008) that are ""led by impartial facilitators who are trained in inspection techniques"" (IEEE, 2017, p. 227) and ""use[] defined team roles and measurement"" (Hamburg and Mogyorodi, 2024)","Static Testing (IEEE, 2021a, p. 21; 2021b, pp. 43-45; 2017, p. 227; Gerrard, 2000a, Fig. 4, p. 12; 2000b, p. 3; Peters and Pedrycz, 2000, Tab. 12.1), Static Analysis (IEEE, 2017, p. 227), Formal Reviews (Hamburg and Mogyorodi, 2024), Role-based Testing (Patton, 2006, p. 95; Peters and Pedrycz, 2000, p. 439), Reviews (Patton, 2006, p. 95), Structure-based Testing (Peters and Pedrycz, 2000, Tab. 12.1), W-Model Testing (Gerrard, 2000a, Fig. 4), Conformance Testing (implied by PMBOK, 2013, p. 533)",,"Related to peer reviews (Hamburg and Mogyorodi, 2024). The code author should be ""a largely silent observer"" who ""may be consulted by the inspectors"" (van Vliet, 2000, p. 415). Roles may be tied to different perspectives, such as designer, implementer, tester (Peters and Pedrycz, 2000, p. 439), user, or product support person (Patton, 2006, p. 95). Changes are made afterwards (van Vliet, 2000, p. 415) and may lead to reinspections (Patton, 2006, p. 95). Could also include inspection of the software product itself, like in the context of usability (Gerrard, 2000a, p. 13)? See ISO 20246"
Installability Testing,"Type (IEEE, 2022, p. 22; 2021c, p. 38, Tab. A.1; 2017, p. 228; implied by its quality)","Testing that determines ""if a test item(s) can be installed, uninstalled/removed and/or upgraded as required in all specified environments"" (IEEE, 2021c, p. 38; similar in ISO/IEC, 2023a; IEEE, 2017, p. 228; OG IEEE, 2013) or a specific one (ISO/IEC, 2023a)","Portability Testing (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1; 2017, p. 228; implied by Kam, 2008, p. 45), Requirements-based Testing (IEEE, 2021c, pp. 37-38), Compatibility Testing (p. 37), Model-based Testing (p. 38), (Replaceability Testing (ISO/IEC, 2023a))",,
Installation Testing,"Level (van Vliet, 2000, p. 439; implied by Washizaki, 2024, p. 5-8)","""System testing conducted in the operational environment of hardware configurations and other operational constraints""; may also verify installation procedures (Washizaki, 2024, p. 5-8) and is usually performed if the system's environment is different from the one in which it was developed (van Vliet, 2000, p. 439)","Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5), Operational Testing? (implied by van Vliet, 2000, p. 439), System Testing? (implied by Gerrard, 2000a, Fig. 5), Online Testing?",,"""Typically observe[s] the newly started server for a while, ensuring that the server doesn't crash or otherwise misbehave"" (Washizaki, 2024, p. 6-10)."
Intake Testing,Approach,,,,
"Integrated System Testing (Firesmith, 2015, p. 24)",Level (inferred from system testing),,"Data Center Testing (Firesmith, 2015, p. 24)",Systems Integration Testing?,
Integration Testing,"Level (IEEE, 2022, pp. 12, 20-22, 26-27; 2021a, pp. 6, 11; 2021b, Fig. 2, pp. 41, 43, 51; 2021c, p. 6; Washizaki, 2025, p. 5-7; Hamburg and Mogyorodi, 2024; Lyu, 1996, p. 532; Black, 2009, pp. 12, 14, 24, 178; Peters and Pedrycz, 2000, Tab. 12.3; van Vliet, 2000, p. 438; Hetzel, 1988, pp. 11, 122, 134, 149, 171; Souza et al., 2017, p. 3; Sakamoto et al., 2013, p. 343; Barbosa et al., 2006, p. 3), Technique (implied by Sharma et al., 2021, pp. 601, 603, 605-606)","Testing that ""verifies the interactions among SUT elements (for instance, components, modules, or subsystems)"" as well as ""external interfaces"" (Washizaki, 2025, p. 5-7; similar in IEEE, 2022, p. 13; 2021a, p. 6; 2021c, p. 6; van Vliet, 2000, p. 438; Sakamoto et al., 2013, p. 343) in a ""progressive"" manner (IEEE, 2017, p. 231)","Dynamic Testing (IEEE, 2021a, pp. 11, 29), Construction Testing (Washizaki, 2025, p. 4-7), Component Integration Testing, System Integration Testing (Hamburg and Mogyorodi, 2024), V-Model Testing (Gerrard, 2000a, p. 9), W-Model Testing (Figs. 3-5), Structure-based Testing (2000b, p. 30), Development Testing (can be) (IEEE, 2021b, p. 42), Independent Tester Testing (can be) (p. 52), Formal Testing (should be) (Hetzel, 1988, p. 133), Systems Testing (sometimes) (p. 122)","Link Testing (implied by Gerrard, 2000a, p. 13)","More effective when automated (Washizaki, 2024, p. 7-14) (this is ""the only way to accomplish this in an effective and efficient manner"" for distributed systems (Sneed and Göschl, 2000, p. 3)), but may be ""complicated and have to be manually executed"" (Gerrard, 2000b, p. 31); it is ""difficult to conduct adequate[ly]"" for web applications (Sakamoto et al., 2013, p. 343). ""Can be performed at each development stage"" (Washizaki, 2025, p. 5-7). IEEE (2017, p. 231) says it can be used when integrating systems; is this captured by ""subsystems"" in the SWEBOK definition? See Patton (2006, p. 109)"
"Integrity Testing (implied by Bas, 2024, pp. 14, 17)","Type (implied by its quality (ISO/IEC, 2023a; IEEE, 2015, p. 8; OG NATO AEP-67))","Testing the ""capability of a product to ensure that the state of its system and data are protected from unauthorized modification or deletion either by malicious action or computer error"" (ISO/IEC, 2023a)","Scenario Testing (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1; 2017, pp. 375, 404), Security Testing (IEEE, 2017, pp. 375, 404-405; 2015, p. 8; OG NATO AEP-67; Washizaki, 2025, p. 5-9), Backup Testing (can be) (Bas, 2024, pp. 14, 17), Dependability Testing (if it exists) (IEEE, 2017, p. 375)",,"Related to immunity testing, if it exists (IEEE, 2017, p. 231)"
Interface Testing,"Level (implied by IEEE, 2017, p. 235; Sakamoto et al., 2013, p. 343; inferred from integration testing), Type (Kam, 2008, p. 45)","Testing that ""aims to verify whether the components' interface provides the correct exchange of data and control information"" (Washizaki, 2024, p. 5-10; similar in IEEE, 2017, p. 235)","SoS Integration Testing (IEEE, 2019b, pp. 48-49), Integration Testing (Hamburg and Mogyorodi, 2024; implied by Gerrard, 2000b, p. 30), Design-based Testing (Kam, 2008, p. 44), System Integration Testing (can be) (p. 48; implied by IEEE, 2021b, p. 44), Specification-based Testing (implied by Washizaki, 2024, p. 5-10)",,
Internationalization Testing,"Type (implied by Firesmith, 2015, p. 55)","Testing that ""anyone on the planet who has the technology can … use"" the software (Gerrard, 2000a, p. 8)","Flexibility Testing (Firesmith, 2015, p. 53), Usability Testing, Functional Testing, Dynamic Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Inclusivity Testing? (if it exists)","Localization Testing? (Gerrard, 2000b, p. 19)","Includes supporting ""local currencies"", ""tax arrangements"", and ""address formats"" (Gerrard, 2000a, p. 8; similar on 2000b, p. 13)"
Interoperability Testing,"Type (IEEE, 2022, p. 22; 2021c, p. 38, Tab. A.1; implied by its quality (ISO/IEC, 2023a); Firesmith, 2015, p. 53)","Testing that aims to evaluate the ""capability of a product to exchange information with other products [or ""systems of different types"" (IEEE, 2017, p. 238)] and mutually use the information that has been exchanged"" (ISO/IEC, 2023a; similar in IEEE, 2021c, p. 38) or that ""a modified system retains [this] capability"" (IEEE, 2017, p. 238)","Compatibility Testing (ISO/IEC, 2023a; IEEE, 2022, p. 3; 2021c, Tab. A.1), Model-based Testing, Requirements-based Testing (p. 38), Conformance Testing (2019b, p. 18), Interface Testing (p. 49)",,"Originally defined in the context of ""two or more systems, products or components"" (ISO/IEC, 2011). Related to functional suitability testing (Hamburg and Mogyorodi, 2024). See ""passive interconnection"" (IEEE, 2017, p. 315; OG IEEE, 2006) potentially?"
"Interrupt-driven Built-In Testing (Firesmith, 2015, p. 31)",Practice?,,"Built-In Testing (Firesmith, 2015, p. 31)","IBIT (Firesmith, 2015, p. 31)",
Isolation Testing,Practice?,"""Testing of individual components in isolation from surrounding components"" (Kam, 2008, p. 45)",Unit Testing?,,"""Surrounding components [may] be[] simulated by stubs and drivers"" (Kam, 2008, p. 45)"
Keyword-driven Testing,"Technique (Hamburg and Mogyorodi, 2024; implied by IEEE, 2016, p. 4)","Testing ""composed from keywords"" (IEEE, 2016, p. 3) ""related to the application being tested"" that ""are interpreted by special supporting scripts that are called by the control script for the test"" (Kam, 2008, p. 45)","Scripted Testing (Hamburg and Mogyorodi, 2024), Automated Testing (can be) (IEEE, 2016, pp. 4, 6; 2022, p. 35), Manual Testing (can be) (IEEE, 2016, pp. 4, 6), Data-driven Testing (Kam, 2008, p. 45)","Action Word-driven Testing (Hamburg and Mogyorodi, 2024), Action-Keyword Testing? (Firesmith, 2015, p. 44)","Similar to data-driven testing (Hamburg and Mogyorodi, 2024). See ISO/IEC/IEEE 29119-5"
Landmark Tours,Practice (inferred from tours),"Tours in which ""testers choose key features, decide on a sequence to visit them, and then explore the application going from landmark to landmark until all of them have been visited"" (IEEE, 2022, p. 34)","Tours, Exploratory Testing (IEEE, 2022, p. 34)",,
Large Scale Integration Testing,"Level (implied by Gerrard, 2000a, p. 13)","Testing that covers ""legacy system and external system integration"" based on dialogs and data validation ""in the context of e-business testing"" (Gerrard, 2000b, p. 30)",Integration Testing,"LSI Testing, LSI (Gerrard, 2000b, p. 30)","Often required due to ""commercial, technical and[/or] logistic constraints"" (Gerrard, 2000b, p. 30)"
"Layer-based Testing (Firesmith, 2015, p. 28)",Approach,,"Unit Testing? (implied by Firesmith, 2015, p. 28)",,
"Legacy System Integration (Testing) (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1)",Level (inferred from system integration testing),,"Functional Testing, Dynamic Testing, Large Scale Integration Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), System Integration Testing, Legacy Testing",,
"Legacy Testing (Firesmith, 2015, p. 34)",Approach,,"Reuse Testing (Firesmith, 2015, p. 34)",,"Often requires ""lower levels of coverage"" when performed ""during maintenance"" (Reid, 1996, p. 3)."
License Compliance Audits,Practice (inferred from audits),"""Audit[s] that reconcile[] license-related information from multiple information sources, such as entitlement consumption against entitlement rights"" (IEEE, 2017, p. 250)","Audits (IEEE, 2017, p. 250)",,
"Lifecycle-based Testing (Firesmith, 2015, p. 29)",Practice?,,,,"""Generally speaking, [the production stage of an SLC] will include the production and testing of the system"" (Washizaki, 2025, p. 10-7)"
"Linear Code Sequence and Jump Testing (Reid, 1996, p. 2)","Technique (Reid, 1996, p. 2)",,"Path Testing (implied by Reid, 1996, p. 3, Fig. 2), Control Flow Testing (implied by p. 3), Structural Testing (implied by p. 6)","Sometimes spelled with an ampersand (Reid, 1996, Fig. 2), LCSAJ Testing (p. 1, Fig. 2)","""It is difficult, if not sometimes impossible, to achieve 100% coverage"" for LCSAJ testing (Reid, 1996, p. 3)"
Linear Scripting,Practice (implied by Scripted Testing),"Scripted testing ""without any control structure in the test scripts"" (Hamburg and Mogyorodi, 2024)","Scripted Testing (Hamburg and Mogyorodi, 2024)",Linear(ly) Scripted Testing?,
"Link Checking (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1)",Approach,"Testing ""the availability of linked objects"" by ""identify[ing] all of the linked objects on a page and … attempt[ing] to download them"" (Gerrard, 2000b, p. 9)","Test Browsing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1, p. 9), Automated Testing (2000a, Tab. 2; 2000b, Tab. 1; implied by p. 32), Smoke Testing, Dynamic Testing (2000a, Tab. 2; 2000b, Tab. 1), Post-Deployment Monitoring (2000b, pp. 32-33), Grey-Box Testing? (Patton, 2006, p. 220), Availability Testing, Web Application Testing",,"May have subapproaches for static, dynamic, or form links (Kam, 2008, p. 8)"
"Link Discoverability (Testing) (inferred from Mesbah and van Deursen, 2009, p. 213)",Approach,,,,
"Link Dependence Transition Relation Testing (Doğan et al., 2014, Tab. 13; OG Peng and Lu, 2011)",Technique (inferred from coverage-based testing),,"Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13), Hyperlink Testing?",,
"Load Balancing Testing (implied by Washizaki, 2024, p. 6-5)",Approach,,,,"May be done by ""us[ing] infrastructure/operations services"" early on in the development process (Washizaki, 2024, p. 6-5)"
Load Testing,"Type (IEEE, 2022, pp. 5, 20, 22; 2017, p. 253; OG IEEE, 2013; Hamburg and Mogyorodi, 2024; implied by Firesmith, 2015, p. 54), Technique (IEEE, 2021c, p. 38-39)","Testing ""conducted to evaluate the behaviour of a test item under anticipated conditions of varying load"" (IEEE, 2022, p. 5; 2017, p. 253; OG IEEE, 2013; similar in 2021c, p. 39; Hamburg and Mogyorodi, 2024); these loads are ""usually between anticipated conditions of low, typical, and peak usage"" (IEEE, 2022, p. 5; 2021c, p. 39; 2017, p. 253; OG IEEE, 2013; Hamburg and Mogyorodi, 2024) or are ""increasing"" (Kam, 2008, p. 45) and may be different numbers of parallel users or transactions (Kam, 2008, p. 45)","Performance-related Testing (IEEE, 2022, p. 22; 2021c, p. 38), Performance Testing (2022, p. 5; 2021c, p. 39; Hamburg and Mogyorodi, 2024; Koomen et al., 2006, p. 50), Reliability Testing (IEEE, 2021c, p. 39; Washizaki, 2025, p. 5-9), Model-based Testing, Requirements-based Testing, Conformance Testing (IEEE, 2021c, p. 38), Performance Efficiency Testing (2017, p. 253; OG IEEE, 2013), Stability Testing, Robustness Testing, Non-functional Testing (Washizaki, 2024, p. 5-9), Negative Testing? (Patton, 2006, p. 86), Capacity Testing (Firesmith, 2015, p. 54), Timing Testing? (implied by Gerrard, 2000a, Tab. 2; 2000b, Tab. 1)",,"May be aided by load generation and/or management (see Hamburg and Mogyorodi, 2024). Patton defines this as running the software with as large of a load as possible (2006, p. 86), and Washizaki (2024, p. 5-9) seems to imply something similar; going past this ""limit"" is the realm of ""stress testing"" (Washizaki, 2024, p. 5-9). Seeks to ""discover problems … or reliability, stability, or robustness violations"" (Washizaki, 2024, p. 5-9). Good to use if the software may have ""a large number of concurrent users"" (IEEE, 2022, p. 45). Captured as part of ""object load and timing"" by Gerrard (2000a, Tab. 2; 2000b, Tab. 1). See ISO 29119-1"
"Local Testing (Firesmith, 2015, p. 42; Jard et al., 1999)",Practice?,,"Synchronous Testing (implied by Jard et al., 1999)",,
Localization Testing,"Type (IEEE, 2022, p. 22; 2021c, p. 38, Tab. A.1)","Testing ""a national or specific regional version of a product"" (IEEE, 2017, p. 253; OG ISO/IEC 2008) ""to determine whether the test item can be understood within the geographical region it is required to be used in"" (IEEE, 2021c, p. 38) and ensure ""that all user messages, prompts and output is [sic] translated correctly and that the functionality delivered to the end user is identical"" (Gerrard, 2000b, p. 19)","Functional Suitability Testing, Accessibility Testing, Portability Testing (IEEE, 2021c, Tab. A.1), Manual Testing (often), Web Application Testing (can be) (Gerrard, 2000b, p. 19)",,"Can include ""analysis of … the user interface and supporting documentation"" (IEEE, 2021c, p. 38). Related to internationalization testing (IEEE, 2017, p. 254; Gerrard, 2000b, p. 19), but can be done ""separately from the translation process"" (IEEE, 2017, p. 254)"
"Loop Testing (Gerrard, 2000a, Fig. 5)",Technique?,"Path testing that focuses on paths containing loops (inferred from Godefroid and Luchaup, 2011, p. 23), ""including nested loops, loops with multiple guards, and arbitrary control-flow graphs with unstructured loops and go-tos"" (p. 24)","Structure-based Testing (Sharma et al., 2021, Fig. 1), Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5), Control Flow Testing? (implied by Godefroid and Luchaup, 2011, p. 24), Performance Testing? (implied by Dhok and Ramanathan, 2016), Unit Testing? (implied by Gerrard, 2000a, Fig. 5)",,"Difficult to automate due to ""virtual call resolution"", reachability conditions, and order-sensitivity (Dhok and Ramanathan, 2016, p. 896). Can be supported by random testing (Dhok and Ramanathan, 2016, p. 896), including fuzz testing (Godefroid and Luchaup, 2011, p. 23), symbolic execution (pp. 23-24, 32), equivalence partitioning (p. 24), and/or the use of loop assertions (p. 23)"
Loopback Testing,Practice?,"""Testing in which signals or data from a test device are input to a system or component, and results are returned to the test device for measurement or comparison"" (IEEE, 2017, p. 257)",,,"Related to ""mechanism loopback"" (IEEE, 2017, p. 270)?"
Low-Level Testing,Approach,"Testing ""with abstract preconditions, input data, expected results, postconditions, and actions (where applicable)"" (Hamburg and Mogyorodi, 2024)",,,"OG definition was about test cases, not a test approach"
"Machine Learning-Assisted Testing (inferred from Moghadam, 2019)",Practice?,,,,
Maintainability Testing,"Type (IEEE, 2022, pp. 5, 22; 2021c, p. 38, Tab. A.1; implied by its quality)","Testing ""conducted to evaluate the degree of effectiveness and efficiency with which a test item may be modified"" (IEEE, 2022, p. 5; similar in ISO/IEC, 2023a) ""by the intended maintainers"" (Hamburg and Mogyorodi, 2024), including through ""corrections, improvements or adaptation of the product to changes in environment, and in requirements and functional specifications"", as well as the ""installation of updates and upgrades"" (ISO/IEC, 2023a)","Model-based Testing, Requirements-based Testing (IEEE, 2021c, p. 38), Regression Testing (can be) (Washizaki, 2024, p. 5-8), Dependability Testing (if it exists) (IEEE, 2017, p. 375)","Serviceability Testing (Kam, 2008, p. 47), Modifiability Testing?","Six categories: corrective, preventive, adaptive, additive, perfective, and emergency (Washizaki, 2024, p. 7-4; ""additive"" and ""emergency"" not present in IEEE, 2021c, p. 38; ""emergency"" added in ISO/IEC/IEEE 14764 and mentioned in IEEE, 2017, p. 156); these may be subapproaches. Can be ""indirectly measured by applying static analysis"" (IEEE, 2021c, p. 38). See IEEE 14764:2022"
Maintenance Testing,"Type (Hamburg and Mogyorodi, 2024)","""Testing the changes to an operational system or the impact of a changed environment to an operational system"" (Hamburg and Mogyorodi, 2024); these changes to the system may be done ""to correct faults, improve performance or others attributes, or adapt to a changed environment"" (IEEE, 2017, p. 261)","Change-Related Testing?, Operational (Acceptance) Testing?",,Related to portability testing and/or retesting?
Malware Scanning,Approach,"The ""detect[ion] and remov[al of] malicious code received at an interface"" (Hamburg and Mogyorodi, 2024)","Static Analysis (Hamburg and Mogyorodi, 2024), Interface Testing (Hamburg and Mogyorodi, 2024)?",,
Manual Testing,"Practice (IEEE, 2022, p. 22)","""Humans performing tests by entering information into a test item and verifying the results"" (IEEE, 2022, p. 6; 2016, p. 3)","Scripted Testing (IEEE, 2022, p. 33)",,"""Inefficient, error prone and non-scalable"" (Washizaki, 2024, p. 6-5)"
"Markov Chain Testing (implied by Kam, 2008, Tab. 1, pp. 21-23)",Technique?,"Testing performed using a Markov chain where transition probabilities are determined by ""web usage model[s] … represent[ing] the possible navigation of the web application"" or are ""uniformly distributed"" ""if there is no usage information available"" (Kam, 2008, p. 22) to find broken links, measure reliability, and evaluate system performance and mean time to failure (p. 23)","Web Application Testing (Kam, 2008, Tab. 1, p. 21; OG [6, 7, 14, 15, 19, 43]), Statistical Testing (Kam, 2008, Tab. 1, pp. 21-22), Specification-based Testing (usually), Non-functional Testing (Tab. 1), Model-based Testing (pp. 21-23), Usage-based Testing (p. 22), Reliability Testing (can be), Performance Testing (can be), Maintenance Testing (can be), Correctness Testing (can be) (p. 23)",,"Is this only applicable in the context of web application testing? My instinct says ""no"""
Mathematical-based Testing,"Practice (IEEE, 2022, pp. 22, 36)","Testing based on ""the test item's required behaviour, input space or output space"" when they ""can be described in sufficient detail"" (IEEE, 2022, p. 36)","Automatic Testing (usually) (IEEE, 2022, p. 36), Structure-based Testing (can be) (Patton, 2006, p. 115)","Math Testing (implied by Hamburg and Mogyorodi, 2024)","Related to computation error testing from Patton (2006, p. 101) or testing formulas and equations (pp. 115-116)? See ISO/IEC/IEEE 29119-4"
MC/DC Testing,"Technique (IEEE, 2022, p. 6; 2021c, pp. 4, 26, Fig. 2; Washizaki, 2025, p. 5-13; Hamburg and Mogyorodi, 2024; Kam, 2008, p. 43; Reid, 1996, p. 2)","Testing based on exercising ""each unique feasible combination of individual Boolean values of conditions within a decision that allows a single Boolean condition to independently affect … [its] outcome"" (IEEE, 2021c, p. 26; similar on p. 4 and in 2022, p. 6; Hamburg and Mogyorodi, 2024; Kam, 2008, p. 43)","Structure-based Testing (IEEE, 2022, pp. 6, 22; 2021c, p. 26, Fig. 2, Hamburg and Mogyorodi, 2024; Kam, 2008, p. 43), Control Flow Testing (IEEE, 2021c, p. 26; Washizaki, 2025, p. 5-13), Model-based Testing (IEEE, 2021c, p. 26), Branch Condition Combination Testing (implied by Hamburg and Mogyorodi, 2024; Kam, 2008, pp. 42-43; Reid, 1996, Fig. 2), Structural Testing (implied by p. 6)","Sometimes spelled without slash (IEEE; 2021c, pp. 4, 26, Tab. A. 1), Modified Condition Decision Testing (IEEE, 2022, p. 6; Washizaki, 2025, p. 5-13; Hamburg and Mogyorodi, 2024; Reid, 1996, p. 2; with slash in IEEE 2021c, pp. 4, 26; Hamburg and Mogyorodi, 2024; Reid, 1996, p. 1), Condition Determination Testing, Modified Multiple Condition Testing (Hamburg and Mogyorodi, 2024)","Requires ""a minimum of N + 1 test coverage items … when there are N conditions within a decision"" (IEEE, 2021c, p. 26) and a minimum of one test case, even if ""there are no decisions in the test item"" (pp. 26, 34). Must be achieved ""for all flight-critical commercial avionics software"" (Reid, 1996, p. 4; OG RTCA SC167/DO-178B), although doing so fully may be ""infeasible"" due to ""strongly coupled conditions"" (Reid, 1996, p. 6)"
Memory Management Testing,"Technique (IEEE, 2021c, p. 38-39), Type (inferred from performance-related testing)","Testing ""aimed at assessing how the test item will perform in terms of the amount (normally maximum) of memory used…, the type of memory … and/or defined levels of memory leakage experienced"", ""typically … in terms of specific operating conditions"" (IEEE, 2021c, p. 39)","Requirements-based Testing (IEEE, 2021c, pp. 38-39), Performance-related Testing, Model-based Testing, Conformance Testing (p. 38), Resource Utilization Testing",,
"Menu Item Testing (implied by IEEE, 2017, p. 444)",Approach,,"Structure-based Testing, System Testing (IEEE, 2021c, p. 444), GUI Testing (implied by p. 496)",,
Metamorphic Testing,"Technique (IEEE, 2022, pp. 6, 22; 2021c, pp. 4, 21, Fig. 2; Washizaki, 2024, p. 5-15; Hamburg and Mogyorodi, 2024; Kanewala and Yueh Chen, 2019, p. 72)","Testing ""based on generating test cases based on existing test cases and metamorphic relations"" (IEEE, 2022, p. 6; 2021c, p. 4; similar on p. 21 and in Hamburg and Mogyorodi, 2024; Kanewala and Yueh Chen, 2019, pp. 67-68)","Specification-based Testing (IEEE, 2022, pp. 6, 22; 2021c, pp. 4, 21, Fig. 2), Model-based Testing (p. 21), Mutation Testing (Washizaki, 2024, p. 5-15), Scripted Testing (can be) (Kanewala and Yueh Chen, 2019, p. 69), Mathematical-based Testing","MT (Hamburg and Mogyorodi, 2024; Kanewala and Yueh Chen, 2019, p. 67)","""There is currently no industry agreed approach to calculating coverage"" and doing so ""as a percentage is often not possible … due to the potentially extremely large number of follow-up test cases that can be derived from a single metamorphic relation"" (IEEE, 2021c, p. 33). Good to use when it is ""difficult to calculate expected results"" (2022, p. 45), when there are a limited number of test cases (e.g., from an experiment that could only be conducted a few times) (Kanewala and Yueh Chen, 2019, pp. 70-72), and/or when working with domain experts who understand the domain and its MRs (p. 70) but not necessarily testing principles (p. 69) or how to identify faults in a program based on its output (p. 71). ""One of the most appropriate and cost-effective testing techniques for scientists and engineers"" (p. 72); helps negate the test oracle (p. 69) and output validation (p. 70) problems. See Chen 2018"
"Method Testing (implied by Doğan et al., 2014, Tab. 11)",Technique (inferred from structure-based testing),,"Structure-based Testing (implied by Doğan et al., 2014, Tab. 11), Control Flow Testing?",,Difference between this and method testing?
"Minimized Testing (implied by IEEE, 2021c, pp. 10-11, 13, 15, 53-55, 62)","Approach (IEEE, 2021c, pp. 10-11, 13, 15, 55)","Testing where ""the minimum number of test cases is derived to cover each … [test coverage item, such as option or mutation for syntax testing (IEEE, 2021c, p. 15) or equivalence partition] at least once"" (p. 11; OG Myers 1979; similar on p. 15) ",,,"""Can lead to such unrealistic test inputs that it can be unlikely that the test item will fail to identify the test input as invalid"", especially for mutations in syntax testing (IEEE, 2021c, p. 15)"
"Migration Testing (Washizaki, 2025, p. 5-8)",Type (inferred from non-functional testing),,"Non-functional Testing, Regression Testing (can be) (Washizaki, 2025, p. 5-8)",Migratability Testing?,
Mixed Entry Table Testing,Technique (inferred from decision table testing),"Testing ""based on exercising decision rules"" (IEEE, 2022, p. 4) in a mixed entry table (IEEE, 2017, p. 278; OG ISO, 1984)","Decision Table Testing (IEEE, 2017, p. 278; OG ISO, 1984)",,
ML Model Testing,"Level (Hamburg and Mogyorodi, 2024)","Testing ""that focuses on the ability of an ML model to meet required ML functional performance criteria and non-functional criteria"" (Hamburg and Mogyorodi, 2024)",Domain-specific Testing?,,"""ML functional performance"" is defined in terms of ""the functional correctness of an ML system"" (Hamburg and Mogyorodi, 2024), which is ambiguous and/or incorrect"
"(Flash) Mob Testing (Firesmith, 2015, pp. 36, 58)","Type (implied by Firesmith, 2015, p. 58)",,"Usability Testing (Firesmith, 2015, p. 58), Group Testing (p. 36)",,
Mobile Testing,"Type (implied by Firesmith, 2015, p. 53)",,"Compatibility Testing (Firesmith, 2015, p. 53)",,
Model Verification,Approach,,"Static Testing (IEEE, 2022, p. 17)",,
Model-based Testing,"Practice (IEEE, 2022, p. 11, Fig. 2; 2021a, p. 5; 2021c, p. viii; 2013, pp. iii, 31), Technique (Souza et al., 2017, p. 3; implied by IEEE, 2017, p. 469)","Testing that uses ""formal or semi-formal representations of the required behaviour of a … [test] item"" to ""generate test cases systematically and automatically"" at ""various levels of abstraction"" (IEEE, 2022, p. 32)","Automated Testing (Firesmith, 2015, p. 44), Web Application Testing (with its subtechniques) (Doğan et al., 2014, Tabs. 8, 22)","Sometimes spelled without a hyphen (Engström and Petersen, 2015, pp. 1-2), MBT (IEEE, 2022, p. 32; Hamburg and Mogyorodi, 2024)","Models ""vary in the degree of model formality"" (such as Agile, semiformal, and formal models) (Washizaki, 2024, p. 1-14) and can be supported by formal specification languages (IEEE, 2022, p. 36). Good potential for automation/generation! See Souza et al., 2017 (term used on p. 3)"
Monkey Testing,"Technique (Washizaki, 2025, p. 5-14; Firesmith, 2015, p. 51)","Testing using ""randomly generated test cases to … cause the program to stop"" (Washizaki, 2025, p. 5-14)","Ad Hoc Testing (Washizaki, 2025, p. 5-14), Random Testing (Firesmith, 2015, p. 51; implied by Washizaki, 2025, p. 5-14), Unscripted Testing (Firesmith, 2015, p. 45), Fuzz Testing",,
Multiplayer Testing,Approach,"""Testing to determine if many players can simultaneously interact with the … game world, … computer-controlled opponents, game servers, and … each other based on the game design"" (Hamburg and Mogyorodi, 2024)",Multi-User Testing,,
Multiple-Hit Decision Table Testing,Technique (inferred from decision table testing),"Testing ""based on exercising decision rules"" (IEEE, 2022, p. 4) in a multiple-hit decision table (IEEE, 2017, p. 285; OG ISO, 1984)","Decision Table Testing (IEEE, 2017, p. 285; OG ISO, 1984)",,
"Multi-User Testing (Gerrard, 2000a, Fig. 5)",Approach,,"Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5)",,
"Mutation Analysis (Peters and Pedrycz, 2000, Tab. 12.1)",Approach,,"Structure-based Testing, Dynamic Testing (Peters and Pedrycz, 2000, Tab. 12.1)",,
Mutation Testing,"Technique (Washizaki, 2025, pp. 5-14 to 5-15; van Vliet, 2000, pp. 428-429; Souza et al., 2017, p. 3; implied by IEEE, 2017, p. 286)","Testing where ""test cases are randomly generated"" or ""specifically designed"" to detect mutants of the SUT (Washizaki, 2025, p. 5-15; similar in Lyu, 1996, p. 533; van Vliet, 2000, pp. 428-429; Sakamoto et al., 2013, p. 352)","Structure-based Testing (Washizaki, 2025, p. 5-15; Lyu, 1996, pp. 532-533), Fault-based Testing (Bourque and Fairley, 2014, p. 4-9; van Vliet, 2000, pp. 428-429), Propagation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3), Web Application Testing (Doğan et al., 2014, p. 181, Tabs. 8, 22), Defect-based Testing (Souza et al., 2017, p. 3), ","Fault Feeding (Doğan et al., 2014, p. 181)","Assumes that ""that more complex but real faults will be found by looking for simple syntactic faults"" and requires ""many mutants … [to] be automatically generated and executed systematically"" (Washizaki, 2024, p. 5-15; similar in van Vliet, 2000, p. 428), such as deleted statements or changed operators (p. 428). Can be used to assist fuzz and metamorphic testing (Washizaki, 2024, p. 5-15) or be supported by tools, such as SimpleJester (Sakamoto et al., 2013, p. 352). Sometimes, ""the original program fail[s], while the modified program yield[s] the right result"" (van Vliet, 2000, p. 432; OG KA85). See Chio (1989)"
Needs-Driven Testing,Approach,"Testing that ""tests why"" the ""system meets stakeholder needs""? (Firesmith, 2015, p. 33)",,,
Negative Testing,"Technique (implied by IEEE, 2021c, pp. 10, 14)","""Testing a component or system in a way for which it was not intended to be used"" (Hamburg and Mogyorodi, 2024; similar in Patton, 2006, p. 84-87), such as for functionality ""not included in the specification"" or ""inputs … that should either be ignored … or cause … an error message"" (IEEE, 2021c, p. 11; similar in Patton, 2006, p. 84-87), using ""at least one input value that the test item should reject as incorrect, ideally with an appropriate error message"" (p. 10). ""The application of boundary value concepts to scenario testing""? (IEEE, 2022, p. 40)","Experience-based Testing (implied by IEEE, 2021c, p. 11; Reid, 1996, p. 7), Security Testing (can be) (Washizaki, 2025, p. 5-9), Forcing Exception Testing? (p. 5-12), Boundary Value Analysis?, Scenario-based Testing? (IEEE, 2022, p. 40), Data Testing (can be) (Patton, 2006, pp. 78-79), State Testing (can be) (pp. 84-87), Robustness Testing?","Invalid Testing (Hamburg and Mogyorodi, 2024; implied by IEEE, 2021c, p. 10), Dirty Testing (Hamburg and Mogyorodi, 2024; Kam, 2008, p. 44), Testing-to-Fail (Patton, 2006, pp. 67, 78, 84-87), Error-Forcing (p. 67)","""Negative test cases can be derived from the state and event combinations that do not appear"" in state models (Washizaki, 2024, p. 1-20)"
Neighborhood Integration Testing,Level (inferred from integration testing),"""Integration testing … [based on] the nodes that connect to a given node"" (Hamburg and Mogyorodi, 2024)","Integration Testing (Hamburg and Mogyorodi, 2024)",,"What does ""node"" mean in this context?"
"Network Admin Testing (Firesmith, 2015, p. 39)",Practice?,,"Operator Testing (Firesmith, 2015, p. 39)",,
"Network Traffic Testing (Firesmith, 2015, p. 24)",Approach,,"Data Center Testing (Firesmith, 2015, p. 24), Network Admin Testing?",,
Neuron Testing,Technique (inferred from coverage-based testing),"Testing based on ""the coverage of activated neurons in the neural network for a set of tests"" (Hamburg and Mogyorodi, 2024)","Coverage-based Testing (Hamburg and Mogyorodi, 2024), ML Model Testing (implied by 2024)",,
"Nominal Testing (mentioned by Ghosh and Voas, 1999, p. 43)",Approach,,,,
Non-functional Testing,"Type (Hamburg and Mogyorodi, 2024; implied by IEEE, 2022, pp. 21-22; 2021b, p. 43)","Testing that ""targets the validation of non-functional aspects (such as performance, usability, or reliability)"" (Washizaki, 2025, p. 5-8; similar in IEEE, 2022, pp. 21-22; Hamburg and Mogyorodi, 2024) that tend to ""constrain the technologies to be used in the implementation"" (Washizaki, 2024, p. 1-4)","Specification-based Testing (Kam, 2008, p. 42), Requirements-based Testing (p. 47), Dynamic Testing (although this seems over-zealous), W-Model Testing (Gerrard, 2000a, p. 9)",,"Hundreds of subapproaches that can be ""performed at all test levels"" (Washizaki, 2025, p. 5-8). ""Nonfunctional requirements"" are also called ""performance attributes"" (IEEE, 2017, p. 293); what does this imply for the relationship between non-functional and performance testing? In constrast to functional testing (IEEE, 2022, p. 21)"
N-Switch Testing,"Technique (inferred from state transition testing and IEEE, 2021c, p. 20)","Testing ""derived to cover valid sequences of N + 1 transitions in the state model"" (IEEE, 2021c, p. 20)","State Transition Testing (IEEE, 2021c, p. 20; Kam, 2008, p. 46)","Multiple Transitions Testing (implied by IEEE, 2021c, p. 20)","See Chow, 1978"
Object-based Testing,Approach,,"Web Application Testing, Object-Oriented Testing, Structure-based Testing, Data Flow Testing, Functional Testing (Kam, 2008, Tab. 1)",,
"Object-Oriented Testing (Doğan et al., 2014, Tab. 1)",Practice?,,,,"See Binder, 1996"
Offline Testing,Practice?,"Testing ""in an environment without external interaction"" (Washizaki, 2024, p. 5-6), or testing ""whereby test cases are generated into a repository for future execution"" (Hamburg and Mogyorodi, 2024)",,,"Sometimes spelled with a hyphen (""off-line""; Washizaki, 2024, p. 5-6)"
"Off-Nominal Testing (mentioned by Ghosh and Voas, 1999, pp. 39, 44)",Approach,,,,
"One-to-One Testing (implied by IEEE, 2021c, pp. 10-11, 13, 15, 51-51, 55, 58, 62)","Approach (IEEE, 2021c, pp. 10-11, 13, 15, 55)","Testing where ""each test case is derived to exercise a specific"" test coverage item, such as an ""option or mutation"" for syntax testing (IEEE, 2021c, p. 15) or equivalence partition (p. 11; OG Myers 1979)",,,
"Ongoing Built-In Testing (Firesmith, 2015, p. 31)",Practice?,,"Built-In Testing (Firesmith, 2015, p. 31), Continuous Testing?","OBIT (Firesmith, 2015, p. 31)",
Online Testing,Practice?,"Testing that ""interacts with the real application environment"" (Washizaki, 2024, p. 5-6), or testing ""whereby test cases are generated and executed simultaneously"" (Hamburg and Mogyorodi, 2024)",,"On-the-Fly Testing (implied by Hamburg and Mogyorodi, 2024)",
OO Web Testing,Approach,,"Web Application Testing, Object-Oriented Testing, Structure-based Testing, Data Flow Testing, Functional Testing (Kam, 2008, Tab. 1)",,
"Open Beta Testing (Firesmith, 2015, p. 58)","Type (implied by Firesmith, 2015, p. 58), Level (inferred from beta testing)",,"Beta Testing (Firesmith, 2015, pp. 39, 58)",,
Open Loop Testing (Control Flow),Technique?,,Loop Testing,,"Not explicitly described by a source (in the context of software control flows); implied by the potential ""closed loop testing"""
Open Loop Testing (Control Systems),Technique?,,"Automated Testing (ideally) (Preuße et al., 2012, p. 1), Safety Testing (can be), Non-functional Testing (can be) (p. 1), Correctness Testing (can be) (p. 6), Model-based Testing (implied by Preuße et al., 2012), Domain-specific Testing?",Sometimes spelled with a hyphen (Preuße et al.),"""Will cause serious problems if the system complexity exceeds the human imagination for thinking about and running test cases"" since ""all possible input information would have to be considered, regardless whether practically relevant"" (Preuße et al., 2012, p. 4). See Pierre et al., (2017; if in scope)"
"Open Source Testing (Firesmith, 2015, p. 34)",Practice?,,"Reuse Testing (Firesmith, 2015, p. 34)",,
Operational (Acceptance) Testing,"Level (IEEE, 2022, p. 22; Firesmith, 2015, p. 30; inferred from acceptance testing)","""Test[ing] to determine the correct installation, configuration and operation of a module and that it operates securely in the operational environment"" (ISO/IEC, 2018) or to ""evaluate a system or component in its operational environment"" (IEEE, 2017, p. 303), particularly ""to determine if operations and/or systems administration staff can accept [it]"" (Hamburg and Mogyorodi, 2024)","Acceptance Testing (IEEE, 2022, p. 22; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 30; LambdaTest, 2024), Non-functional Testing (LambdaTest, 2024), Procedure Testing? (IEEE, 2021c, p. 40), Reliability Testing (although this seems wrong) (Bourque and Fairley, 2014, p. 4-6)","Production Acceptance Testing (Hamburg and Mogyorodi, 2024), Operating Testing, Operational Readiness Testing, ORT, (LambdaTest, 2024), OAT, OT (Firesmith, 2015, p. 30)","Ensured by the use of TDD and ATDD (Washizaki, 2025, p. 6-9). ""Operational testing"" and ""operational acceptance testing"" are treated as synonyms in this glossary, although there is sometimes a distinction (Firesmith, 2015, p. 30)"
Operational Effectiveness Testing,"Level (Firesmith, 2015, p. 30; inferred from operational testing)",,"Operational Testing (Firesmith, 2015, p. 30)",,
Operational Profile Testing,Approach,"""Testing using a model of system operations (short duration tasks) and their probability of typical use"" (Kam, 2008, p. 46; OG Musa (et al., 1987?))","Reliability Testing (Lyu, 1996, pp. 534-535), Statistical Testing (Kam, 2008, p. 46; OG Musa), Specification-based Testing (can be) (Lyu, 1996, p. 535), Customer Acceptance Testing (implied by p. 535), Usage-based Testing, Scenario Testing?",,
Operational Suitability Testing,"Level (Firesmith, 2015, p. 30; inferred from operational testing)",,"Operational Testing (Firesmith, 2015, p. 30)",,
"Operations Organization Testing (Firesmith, 2015, p. 37)",Practice?,,"Organization-based Testing (Firesmith, 2015, p. 37), Operational Testing?",,
"Operator Testing (Firesmith, 2015, p. 39)",Practice?,,"Role-based Testing (Firesmith, 2015, p. 39)",,
Organization-based Testing,"Process (implied by IEEE, 2022, p. 24)",,Role-based Testing?,Role-based Testing?,
Orthogonal Array Testing,"Technique (Washizaki, 2025, p. 5-11; implied by Mandl, 1985, p. 1054)","Testing orthogonal aspects of ""the state space[, which] is the Cartesian product of the ranges"" of its relevant ""factors"" as long as they can be ""select[ed] … independently of each other and in any order"" (Mandl, 1985, p. 1054). These factors can be based on values (index factors) or ""the shape of the construct"" (contents factors) (p. 1056)","Combinatorial Testing (Washizaki, 2025, p. 5-11; Valcheva, 2013, p. 467; implied by p. 473; Yu et al., 2011, p. 1573; Mandl, 1985, p. 1055), Statistical Testing (Valcheva, 2013, p. 467; Tsui, 2007, p. 44; implied by Mandl, 1985, p. 1056), Mathematical-based Testing (Valcheva, 2013, p. 467), Software Interaction Testing (p. 468), Equivalence Partitioning (Mandl, 1985, p. 1055), Compiler Testing (can be, such as in ""the Ada Compiler Validation Capability (ACVC) test suite"") (p. 1054)","OAT (Washizaki, 2025, pp. 5-1, 5-11; implied by Valcheva, 2013, pp. 467, 473; Yu et al., 2011, pp. 1573-1577, 1580), Orthogonal-Latin-Squares Method (Mandl, 1985, p. 1054; implied by Valcheva, 2013, pp. 469-470), Pairwise Testing (implied by p. 473)","""Yields the informational equivalent of exhaustive testing at a fraction of the cost"" (Mandl, 1985, p. 1054) (reduces the number of test cases with ""k factors with n levels each"" (p. 1055) from n^k to n^2 (Valcheva, 2013, p. 468; Mandl, 1985, pp. 1055-1056)) and a higher level of confidence than random testing (p. 1055)"
"OT Organization Testing (Firesmith, 2015, p. 37)",Practice?,,"Independent Test Organization Testing (Firesmith, 2015, p. 37), Operational Testing? (Firesmith, 2015, p. 30)",,
"Output Domain Testing (Peters and Pedrycz, 2000, Fig. 12.2)",Approach,,"Specification-based Testing (implied by Peters and Pedrycz, 2000, Fig. 12.2)",,
"Outside-In Testing (Firesmith, 2015, p. 28)",Level (inferred from integration testing),,Integration Testing? (see bottom-up and top-down testing),,
Outsourced Testing,Practice?,"""Testing performed by people who are not co-located with the project team and are not fellow employees"" (Hamburg and Mogyorodi, 2024)",Independent Tester Testing?,,"Also mentioned by Firesmith (2015, p. 41)"
"Page Testing (Doğan et al., 2014, Tab. 13; OG Ricca and Tonella, 2001; OG 2002)",Technique (inferred from coverage-based testing),"Testing in which ""every page in the SUT is visited at least once"" (Doğan et al., 2014, Tab. 13; OG Ricca and Tonella, 2001; OG 2002)","Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13), Control Flow Testing?",,
Pair Testing,"Technique (Washizaki, 2025, p. 5-14)","Testing in which ""two persons, e.g. two testers, a developer and a tester, or an end-user and a tester, work[] together to find defects"" (Kam, 2008, p. 46), usually structured so that ""one generates and runs the test cases[ and] the other observes and analyzes the testing process"" (Washizaki, 2025, p. 5-14) ""shar[ing] one computer and trad[ing] control of it"" (Kam, 2008, p. 46)","Ad Hoc Testing (Washizaki, 2025, p. 5-14), Group Testing (Firesmith, 2015, p. 36), Embedded Tester Testing (p. 39), Developer Testing (implied by Gerrard, 2000a, p. 11)","Buddy Testing (Washizaki, 2025, p. 5-14)","""Allows for generating test cases with broad and better test coverage"" (Washizaki, 2025, p. 5-14)"
Pairwise Integration Testing,Level (inferred from integration testing),"""Integration testing that targets pairs of components that work together as shown in a call graph"" (Hamburg and Mogyorodi, 2024)","Integration Testing (Hamburg and Mogyorodi, 2024)",,
Pairwise Testing,"Technique (IEEE, 2022, pp. 7, 22; 2021c, pp. 2, 16, Fig. 2; Washizaki, 2025, p. 5-11; Hamburg and Mogyorodi, 2024)","Testing that covers ""all possible pairs"" of some set of ""unique pairs of P-V pairs, where each P-V pair within the pair is for a different test item parameter"" (IEEE, 2021c, p. 16)","Combinatorial Testing (IEEE, 2022, pp. 7, 22; 2021c, pp. 2, 16, Fig. 2; Washizaki, 2024, p. 5-11), Specification-based Testing (IEEE, 2022, p. 7; Hamburg and Mogyorodi, 2024), Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, p. 3; 2021c, p. 6), All Combinations Testing (p. 16), t-wise Testing (Washizaki, 2025, p. 5-11), Orthogonal Array Testing (p. 5-11; Valcheva, 2013, p. 473; implied by Mandl, 1985, p. 1055)","Sometimes spelled with a hyphen (IEEE, 2021c, pp. 2, 16, Fig. 2; Valcheva, 2013, p. 473), All Pairs Testing (IEEE, 2021c, p. 16)","""The most popular form of combinatorial testing"" (IEEE, 2022, p. 7)"
"Partial Regression Testing (Firesmith, 2015, p. 34)",Approach,,"Regression Testing (Firesmith, 2015, p. 34)","Selective Retesting? (implied by IEEE, 2017, p. 372; Washizaki, 2024, p. 5-8)",
Password Cracking,Approach,"""Security attack[s] recovering secret passwords stored in a computer system or transmitted over a network"" (Hamburg and Mogyorodi, 2024)","Security Attacks (Hamburg and Mogyorodi, 2024), Security Testing (implied by ""transmitted over a network"" in Hamburg and Mogyorodi, 2024), Network Admin Testing?",,See NIST.IR.7298
Path Testing,"Technique (Washizaki, 2025, p. 5-13; Kam, 2008, p. 46)","Testing that ""aims to execute all [""or selected"" (IEEE, 2017, p. 316)] entry-to-exit control flow paths in a SUT's control flow graph"" (Washizaki, 2025, p. 5-13; similar in IEEE, 2017, p. 316; Patton, 2006, p. 119)","Control Flow Testing (Washizaki, 2025, p. 5-13; Reid, 1996, p. 3; implied by Hamburg and Mogyorodi, 2024), Exhaustive Testing (Peters and Pedrycz, 2000, pp. 466-467, 476; implied by Patton, 2006, pp. 120-121), Propagation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3), Structure-based Testing (Sharma et al., 2021, Fig. 1; Kam, 2008, p. 46; implied by Peters and Pedrycz, 2000, Tab. 12.1), W-Model Testing (Gerrard, 2000a, Fig. 5), Dynamic Testing (Fig. 5; implied by Peters and Pedrycz, 2000, Tab. 12.1), Structural Testing (implied by Reid, 1996, p. 6)","Path-based Testing (implied by Peters and Pedrycz, 2000, Tab. 12.1), All-Paths Testing? (van Vliet, 2000, p. 421, Fig. 13.17), Exhaustive Testing (incorrectly) (p. 421), All Round-Trip Paths Testing? (Kam, 2008, p. 15)","""Exhaustive path testing is generally not feasible because of loops"" (Washizaki, 2025, p. 5-13; similar in Peters and Pedrycz, 2000, p. 473-476; van Vliet, 2000, p. 421; Reid, 1996, p. 3) (which ""often result[] in an infinite number of possible paths"" (van Vliet, 2000, p. 421)), the exponential increase of possible paths with more branches (Peters and Pedrycz, 2000, p. 473-476; van Vliet, 2000, p. 421), potentially unreachable code (p. 421), and the ""significant problem"" of infeasible paths (Washizaki, 2024, p. 5-5; similar in Peters and Pedrycz, 2000, p. 439; van Vliet, 2000, p. 421; Reid, 1996, p. 3), although infeasible code may be excluded from coverage calculations with adequate justification and documentation (p. 3). Because of these limitations, exhaustive path testing is usually ""limited to a few functions with life criticality features (medical systems, real-time controllers)"" (Peters and Pedrycz, 2000, p. 481; OG Miller et al., 1994). However, the number of paths to test can be bounded based on the code's structure and be approached by dividing the system into subgraphs and computing the bounds of each individually (Peters and Pedrycz, 2000, p. 471-473); this bound should be at least ""the number of linearly independent paths through a program's source code"" (Washizaki, 2024, p. 4-2). Possibly involves ""path analysis"""
Patterns-based Testing,"Technique (Firesmith, 2015, p. 46)",,,,
Peer Reviews,Approach,"""Review[s] of work products performed by peers [with similar abilities (Hamburg and Mogyorodi, 2024)] during development of the work products to identify defects for removal"" (Washizaki, 2024, p. 12-13; OG [14], might be IEEE, 2017, p. 317?)","Reviews (IEEE, 2017, p. 317; Washizaki, 2024, p. 12-13; Hamburg and Mogyorodi, 2024; Patton, 2006, p. 94), Informal Testing (p. 94), Static Analysis? (Washizaki, 2024, p. 12-13)","Buddy Reviews (Patton, 2006, p. 94)","""Often performed during development of the work products to identify defects for removal"", ""increase the quality of the work product"", and ""reduce cost by fixing defects as soon as possible"" (IEEE, 2017, p. 317). Can be done through pair programming (Washizaki, 2024, p. 12-14), a group of two or three people going through code that one of them wrote (Patton, 2006, p. 94), or by randomly distributing ""a 'best' program and one of lesser quality"" from each person to pairs in the group to be assessed, then returning feedback anonymously (van Vliet, 2000, p. 414). See ISO 20246"
Penetration Testing,"Type (IEEE, 2021b, pp. 41, 43; implied by Firesmith, 2015, p. 57; inferred from security testing), Technique (IEEE, 2021c, p. 40; Hamburg and Mogyorodi, 2024)","Testing that ""tests a system in its final production environment"" (Washizaki, 2024, p. 13-5) by ""exploit[ing] security vulnerabilities (known or unknown)"" (Hamburg and Mogyorodi, 2024) and ""involves attempted access to a test item (including its functionality and/or private data) by a tester that is mimicking the actions of an unauthorised user"" (IEEE, 2021c, p. 40), such as ""submit[ting] malformed, malicious and random data to [its] entry points"" (Washizaki, 2024, p. 13-5)","Security Testing (IEEE, 2021c, p. 40; Washizaki, 2024, p. 13-4; Firesmith, 2015, p. 57; Gerrard, 2000b, pp. 28-29; implied by Hamburg and Mogyorodi, 2024), Privacy Testing (IEEE, 2021c, p. 40), Dynamic Testing, Specification-based Testing (Patton, 2006, p. 88), Attacks (Gerrard, 2000b, p. 28), Non-functional Testing (implied by IEEE, 2021b, p. 43), Web Application Testing (implied by Doğan et al., 2014, p. 194), Operational Testing, Online Testing?","Ethical Hacking Testing (Washizaki, 2024, p. 13-4), Ethical Hacking (Gerrard, 2000b, p. 28)","Should be conducted by security experts (Washizaki, 2024, p. 13-5) and may be done in a pre-production and/or production environment (IEEE, 2021b, p. 42). Related to fuzz testing (Washizaki, 2024, p. 13-5)?"
Performance Efficiency Testing,Type (inferred from performance testing),"Testing to evaluate the ""capability of a product to perform its functions within specified time and throughput parameters and be efficient in the use of resources under specified conditions"" (ISO/IEC, 2023a; similar but less specific in IEEE, 2017, p. 319)","Reliability Testing (ISO/IEC, 2023a), Performance-related Testing (IEEE, 2021c, Tab. A.1), Performance Testing (implied by IEEE, 2017, p. 319), Efficiency Testing? (ISO/IEC, 2023a), Dependability Testing (if it exists) (ISO/IEC, 2023a)",,"Resources can include devices, configurations, energy, or materials (ISO/IEC, 2023a). Related to the idea of ""performance deficiency"" (IEEE, 2017, p. 319)"
Performance Testing,"Type (IEEE, 2022, pp. 7, 22, 26-27; 2021a, pp. 2, 8, 11; 2021b, pp. 41, 43; 2021c, p. 7; Koomen et al., 2006, pp. 9, 50; implied by Firesmith, 2015, p. 53), Technique (IEEE, 2021c, p. 38-39)","Testing ""conducted to evaluate the degree to which a test item accomplishes its designated functions within given constraints of time and other resources"" (IEEE, 2022, p. 7; 2021a, p. 2; 2017, p. 320; similar in Hamburg and Mogyorodi, 2024; OG ISO 25010?; Moghadam, 2019, p. 1187) or ""under a 'typical' load"" (IEEE, 2021c, p. 39), focused on ""measuring the performance metrics"" (Moghadam, 2019, p. 1187; similar in Hamburg and Mogyorodi, 2024) (such as the ""system's capacity for growth"" (Gerrard, 2000b, p. 23)), ""detecting the functional problems appearing under certain execution conditions"" (Moghadam, 2019, p. 1187), and ""detecting violations of non-functional requirements under expected and stress conditions"" (Moghadam, 2019, p. 1187; similar in Washizaki, 2025, pp. 5-8 to 5-9)","Performance-related Testing (IEEE, 2022, p. 22; 2021c, p. 38), Dynamic Testing (2021a, pp. 11, 29-30; Gerrard, 2000a, Fig. 5, Tab. 2; 2000b, Tab. 1), Requirements-based Testing (IEEE, 2021c, p. 38; Moghadam, 2019, p. 1187), Model-based Testing, Conformance Testing (IEEE, 2021c, p. 38), Non-functional Testing (Washizaki, 2024, p. 5-8; Bertolino et al., 2019, p. 93:19; Moghadam, 2019, p. 1187; Kam, 2008, pp. 46-47; Gerrard, 2000a, Tab. 2; 2000b, Tab. 1, p. 22; implied by IEEE, 2021b, p. 43), Automated Testing (Dhok and Ramanathan, 2016, p. 895; Gerrard, 2000a, Tab. 2; 2000b, Tab. 1; can be in IEEE, 2021b, p. 10; implied by p. 32), Performance Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Post-Deployment Monitoring (2000b, p. 32), Dynamic Analysis, Static Analysis (Dhok and Ramanathan, 2016, p. 895), Experience-based Testing (can be) (IEEE, 2022, p. 4; 2021c, p. 4), Regression Testing (can be) (ISO/IEC, 2014c), Cloud Testing (can be) (Washizaki, 2025, p. 5-27), Web Application Testing (can be) (Gerrard, 2000b, p. 3)","Performance-related Testing (Moghadam, 2019, p. 1187), Performance Efficiency Testing (implied by Hamburg and Mogyorodi, 2024)","""Performance, load and stress testing might considerably overlap in many areas"" (Moghadam, 2019, p. 1187). ""Performance issues are hard to detect in-house during testing and usually manifest in the field"" (Dhok and Ramanathan, 2016, p. 895; OG [28]) so this may be performed in a pre-production environment (IEEE, 2021b, p. 42), by ""us[ing] infrastructure/operations services"" early on in the development process (Washizaki, 2024, p. 6-5), or by using automated test tools (Gerrard, 2000b, p. 24). Performance issues include ""repetitive computations, redundant loops, object bloat, latent performance bugs, and performance issues in clouds and smart phones"" (Dhok and Ramanathan, 2016, p. 895) and may be specific to ""real-time constraints"" (IEEE, 2022, p. 43). Requires well-defined requirements, stable software, production hardware, and a controlled test environment, and estimating an ""upper limit"" for web application testing isn't feasible (p. 23); in this case, testing with test drivers may be sufficient (p. 24). Helps ""identify weak points of a software system and quantify its shortcomings"" (Peters and Pedrycz, 2000, p. 447). Difference between this and performance efficiency testing? Related to efficiency testing (Kam, 2008, p. 46)"
Performance-related Testing,"Type (IEEE, 2022, p. 22; 2021c, p. 38, Tab. A.1)","Testing ""to determine whether a test item performs as required when it is placed under various types and sizes of 'load'"" (IEEE, 2021c, p. 38)",,,
"Periodic Built-In Testing (Firesmith, 2015, p. 31)",Practice?,,"Built-In Testing (Firesmith, 2015, p. 31)","PBIT (Firesmith, 2015, p. 31)",
"Personalization Testing (Firesmith, 2015, p. 53)","Type (implied by Firesmith, 2015, p. 53)",,"Flexibility Testing (Firesmith, 2015, p. 53)",,
"Perturbation Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Infection-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",,
Pharming,Approach,"""Security attack[s] intended to redirect a website's traffic to a fraudulent website without the user's knowledge or consent"" (Hamburg and Mogyorodi, 2024)","Security Attacks (Hamburg and Mogyorodi, 2024)",,
Physical Configuration Audits,Practice (inferred from audits),"""Audit[s] conducted to verify that a configuration item, as built, conforms to the technical documentation that defines it"" (IEEE, 2017, p. 322)","Audits (IEEE, 2017, p. 250), Configuration Testing","PCAs (IEEE, 2017, pp. 317, 322; 2015, p. 10)",
Player Perspective Testing,Level?,"""Testing done by testers from a player's perspective to validate player satisfaction"" (Hamburg and Mogyorodi, 2024)","Satisfaction Testing (if it exists) (Hamburg and Mogyorodi, 2024)",,
Playtesting,Level?,"""Testing of a game [performed] by players to identify failures and gather feedback"" (Hamburg and Mogyorodi, 2024)","Ad Hoc Testing (Hamburg and Mogyorodi, 2024), User as Tester Testing",,
Portability Testing,"Type (IEEE, 2022, pp. 7, 22; 2021c, p. 39; Koomen et al., 2006, p. 9; implied by its quality (ISO/IEC, 2023a; Hamburg and Mogyorodi, 2024))","Testing to evaluate the ""capability of a product to be adapted to changes in its requirements, contexts of use, or system environment"" (ISO/IEC, 2023a; similar in IEEE, 2022, p. 7; 2017, pp. 184, 329; Hamburg and Mogyorodi, 2024), such as being ""transferred from one hardware, software or other operational or usage environment to another"" or ""alter[ing] the configuration of the existing environment"" (IEEE, 2021c, p. 39)","Model-based Testing, Requirements-based Testing (IEEE, 2021c, p. 39)","Flexibility Testing (ISO/IEC, 2023a), Transportability Testing (implied by IEEE, 2017, p. 329)","Related to co-existence and installability testing, as well as adaptability and replaceability testing (if they exist) (Hamburg and Mogyorodi, 2024) and potentially compatibility testing, and can improve user assistance and interaction capability (ISO/IEC, 2023a) ""Can be measured either as the extent to which a product can be used by additional types of users to achieve additional types of goals … or by a capability to be modified to support adaptation for new types of users"", etc. (ISO/IEC, 2011)"
Positive Testing,"Technique (implied by IEEE, 2021c, p. 10)","Testing that uses ""input values that the test item should accept as correct"" (IEEE, 2021c, p. 10)",,"Valid Testing (implied by IEEE, 2021c, p. 10), Testing-to-Pass (Patton, 2006, p. 66)","Should be done before more complicated, unusual negative testing (Patton, 2006, p. 66)"
Post-Deployment Monitoring,"Level (implied by Gerrard, 2000a, p. 13)","""Monitor[ing a] … site in production"" (Gerrard, 2000b, p. 32; similar in 2000a, p. 15), including the server hardware, CGI programs, and links (2000b, p. 32)","Automated Testing (Gerrard, 2000a, pp. 13, 15), Performance Testing (implied by p. 15), Regression Testing",,"May include the use of ""retain[ed] automated tests"" (Gerrard, 2000a, p. 13; similar in 2000b, p. 34), remote monitoring services, and/or consultants (2000b, p. 34)"
Post-Release Testing,"Type (Hamburg and Mogyorodi, 2024)","""Testing to ensure that the release is performed correctly and the application can be deployed"" (Hamburg and Mogyorodi, 2024)","Online Testing?, Operational Testing?",,
"Power Testing (implied by IEEE, 2022, p. 43)",Approach,"Testing ""based on power consumption and battery failure"" (IEEE, 2022, p. 43)","Domain-specific Testing, Performance-related Testing?",,
"Power-Up Built-In Testing (Firesmith, 2015, p. 31)",Practice?,,"Built-In Testing (Firesmith, 2015, p. 31)","PupBIT (Firesmith, 2015, p. 31)",
"Prime Contractor Testing (Firesmith, 2015, p. 37)",Practice?,,"Development Organization Testing (Firesmith, 2015, p. 37)",,
"Prime Path Testing (implied by Doğan et al., 2014, p. 184)",Technique?,,"Model-based Testing (Doğan et al., 2014, p. 184)",,"See Sakamoto et al., (2013)"
Prioritization Testing,Practice?,"Testing that ""schedule[s] test cases to increase the rate [and likelihood] of fault detection, … the coverage of code under test, and … reliability"" (Washizaki, 2024, p. 5-8)",,,"What does ""schedule"" mean in this context?"
Privacy Testing,"Technique (IEEE, 2021c, p. 40), Type (inferred from security testing)","Testing that ""assess[es] the security and privacy of users' personal data to prevent local attacks"" (Washizaki, 2024, p. 5-10) and ""involves attempted access to private data and verification of the audit trail (i.e. trace) that is left behind when users access private data"" (IEEE, 2021c, p. 40)","Security Testing (IEEE, 2021c, p. 40), Compliance Testing (implied by Bas, 2024, pp. 26-28)",,"Assesses policies and profile/data management (Washizaki, 2024, p. 5-10)"
"Probable Correctness Testing (Peters and Pedrycz, 2000, Fig. 12.4)",Approach,,"Error-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.4)",Relative Correctness Testing?,
Procedure Testing,"Type (IEEE, 2022, pp. 7, 22; 2021c, p. 39, Tab. A.1; 2017, p. 337; OG IEEE, 2013), Technique (implied by Firesmith, 2015, p. 47)","Testing ""conducted to evaluate whether procedural instructions for interacting with a test item or using its outputs meet user requirements and support the purpose of their use"" (IEEE, 2022, p. 7; 2017, p. 337; OG 2013; similar in 2021c, p. 39)","Functional Suitability Testing (IEEE, 2022, p. 7; 2017, p. 337; OG 2013), Requirements-based Testing (2022, p. 7; 2021c, p. 39; 2017, p. 337; OG 2013), Model-based Testing (2021c, p. 39), Specification-based Testing (implied by Firesmith, 2015, p. 47)",,"Some of this is out of scope, such as user guides or manuals, if they are not part of the system (e.g., in ""tutorial files"") (IEEE, 2021c, p. 39)"
Process-Driven Scripting,"Technique (Hamburg and Mogyorodi, 2024)","Testing ""where scripts are structured into scenarios which represent use cases of the software under test"" and ""can be parameterized with test data"" (Hamburg and Mogyorodi, 2024)","Scripted Testing, Scenario Testing (Hamburg and Mogyorodi, 2024)",,
"Processor-in-the-Loop Testing (Firesmith, 2015, p. 23)",Level (inferred from system testing),,"System Testing (Firesmith, 2015, p. 23)","PIL Testing (Firesmith, 2015, p. 23)",
"Product Lines Testing (Doğan et al., 2014, Tab. 1)",Approach,,,,"See Neto et al., (2012; 2011a; 2011b) and Engström and Runeson (2011)"
Production Acceptance Testing,"Level (Firesmith, 2015, p. 30; inferred from acceptance testing)",,"Acceptance Testing (Firesmith, 2015, p. 30)","Operational Acceptance Testing (Hamburg and Mogyorodi, 2024), PAT (Firesmith, 2015, p. 30)",
Production Verification Testing,"Level (IEEE, 2022, p. 22; 2021b, pp. 41, 43, 51; inferred from acceptance testing)",,"Acceptance Testing (IEEE, 2022, p. 22), Independent Tester Testing (can be) (2021b, p. 52), Online Testing?, Operational Testing?",Production Acceptance Testing?,"May be done in a production environment (IEEE, 2021b, p. 42)"
Project-based Testing,Process,"Testing designed ""to support the management of testing and to support dynamic testing"" (IEEE, 2022, p. 25)","Dynamic Testing, Lifecycle-based Testing (usually), Compliance Testing (can be) (IEEE, 2022, p. 25)",,
"Prognostics and Health Management (Firesmith, 2015, p. 31)",Practice?,,"Ongoing Built-In Testing, Self-Testing (can be) (Firesmith, 2015, p. 31)","PMH (Firesmith, 2015, p. 31)",
"Programmer Testing (Firesmith, 2015, p. 39)",Practice?,,"Developer Testing (Firesmith, 2015, p. 39)",Developer Testing?,
Proofs of Correctness,"Technique (IEEE, 2017, p. 355; van Vliet, 2000, p. 418), Artifact (IEEE, 2017, p. 355)","""Highly formal methods of logic"" (Peters and Pedrycz, 2000, p. 438) ""used to prove mathematically that a computer program satisfies its specified requirements"" (IEEE, 2017, p. 355) or that ""an equivalence between the program and its specification"" exists (Peters and Pedrycz, 2000, p. 485)","Manual Testing, Static Analysis, Specification-based Testing, Formal Testing (van Vliet, 2000, p. 418), Structure-based Testing, Static Testing (implied by Peters and Pedrycz, 2000, Tab. 12.1), Correctness Testing, Mathematical-based Testing","Correctness Proofs (van Vliet, 2000, p. 418), Program Proving (excludes artifact) (implied by Peters and Pedrycz, 2000, Tab. 12.1)","Not often used; requires a formal specification and its value is ""sometimes disputed"" (van Vliet, 2000, p. 418), although this may be beneficial for Drasil"
"Proofs of Partial Correctness (inferred from partial correctness (IEEE, 2017, p. 314) and proofs of correctness)","Technique, Artifact (IEEE, 2017, p. 355)","""Formal technique[s] used to prove mathematically"" (IEEE, 2017, p. 355) ""that a program's output assertions follow logically from its input assertions and processing steps"" (p. 314)","Proofs of Correctness, Formal Testing, Mathematical-based Testing, Static Testing?",,Difference between this and total correctness?
"Proofs of Total Correctness (inferred from total correctness (IEEE, 2017, p. 480) and proofs of correctness)","Technique, Artifact (IEEE, 2017, p. 355)","""Formal technique[s] used to prove mathematically"" (IEEE, 2017, p. 355) ""that a program's output assertions follow logically from its input assertions and processing steps"" (p. 480)","Proofs of Correctness, Formal Testing, Mathematical-based Testing, Static Testing?",,Difference between this and partial correctness?
"Propagation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Implementation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",,
"Protection System Testing (Forsyth et al., 2004)",Approach,,"System Testing, Domain-specific Testing?",,"Often performed using ""real time simulators"" (Forsyth et al., 2004, p. 332)"
Qualification Operational Testing,Level (inferred from operational testing),,"Operational Testing (Firesmith, 2015, p. 30), Qualification Testing?","QOT (Firesmith, 2015, p. 30)",Difference between this and qualification testing?
Qualification Testing,Level?,"""Testing … to demonstrate that a software product meets its specifications and is ready for use in its target environment or integration with its containing system"" (IEEE, 2017, p. 360; OG ISO/IEC, 2008)",Acceptance Testing?,"Acceptance Testing (Bourque and Fairley, 2014, p. 4-6), Operational (Acceptance) Testing?","""Conducted by the developer and witnessed by the acquirer (as appropriate)"" (IEEE, 2017, p. 360). Including ""software"" in its definition may be wrong, since hardware qualification testing exists (Firesmith, 2015, p. 21). Difference between this and qualification operational testing?"
Quality Assurance,Approach,"A ""part of quality management focused on providing confidence that quality requirements will be fulfilled"" (IEEE, 2015, p. 7; OG ISO, 2005)","Conformance Testing (implied by PMBOK, 2013, pp. 244, 535), Requirements-based Testing","QA (IEEE, 2022, p. 16; 2017, pp. 359-360; 2015, p. 10; implied by PMBOK, 2013, p. 476)","Implied to be a test approach in (Washizaki, 2024, p. 5-4)"
Quick Testing,"Technique (Washizaki, 2025, p. 5-14)","Testing ""in which a very small test suite is selected and executed to swiftly identify critical issues in the SUT … [and] detect[] faults early in the development process"" (Washizaki, 2025, p. 5-14)","Ad Hoc Testing (Washizaki, 2025, p. 5-14)",,
Race Condition Testing,Type (inferred from robustness testing),"Testing a system's ability to handle ""two or more events lin[ing] up"" disruptively when it ""didn’t expect to be interrupted in the middle of its operation"" (Patton, 2006, p. 85)","Negative Testing (Patton, 2006, p. 85), Robustness Testing (p. 86)",,
Random Testing,"Technique (IEEE, 2022, pp. 7, 22, 36; 2021c, pp. 5, 20, Fig. 2; Washizaki, 2024, p. 5-12; Hamburg and Mogyorodi, 2024; Firesmith, 2015, pp. 46, 51)","Testing ""based on generating test cases to exercise randomly selected test item inputs"" (IEEE, 2022, p. 7; 2021c, p. 5) from ""the set of all possible input values"" based on an input distribution, including the normal distribution, a uniform distribution (p. 21), or an operational profile (p. 21; Hamburg and Mogyorodi, 2024)","Specification-based Testing (IEEE, 2022, pp. 7, 22; 2021c, pp. 5, 20, Fig. 2; Washizaki, 2024, p. 5-12; Hamburg and Mogyorodi, 2024; Peters and Pedrycz, 2000, Tab. 12.1), Mathematical-based Testing (IEEE, 2022, p. 36), Model-based Testing (IEEE, 2021c, p. 21), Operational Profile Testing (p. 21; Hamburg and Mogyorodi, 2024), Usage-based Testing (Washizaki, 2024, p. 5-15), Dynamic Testing (Peters and Pedrycz, 2000, Tab. 12.1)","Cat on the Keyboard Testing (Firesmith, 2015, pp. 46, 51), Randomized Testing (Doğan et al., 2014, p. 192; OG Halfond and Orso, 2007)","""There is currently no industry agreed approach to calculating coverage"" (IEEE, 2021c, p. 33). Often used in automated testing (Washizaki, 2024, p. 5-12) or to support non-functional testing, such as performance testing (Kam, 2008, p. 46) or reliability testing (Kam, 2008, p. 46; van Vliet, 2000, p. 439). Implied to be related to statistical testing (Washizaki, 2024, p. 5-15). ""Less effective in detecting malfunctions and yields … less information when it does"" (Mandl, 1985, p. 1058). See Craig and Jaskiel (2002); Kaner (1988)"
"Random Walk Testing (Kam, 2008, p. 12; OG [18])",Technique (inferred from path testing),"Testing that ""ensures arbitrary path selection with the same probability to be chosen"" (Kam, 2008, p. 12)","Path Testing, Web Application Testing (implied by Kam, 2008, p. 12)",,
Rapid Prototyping Testing,Practice?,"Testing performed on prototypes developed ""early in the development process to permit early feedback and analysis"" (IEEE, 2017, p. 365)","Lifecycle-based Testing, At-the-Beginning Testing?",,
Reactive Testing,Practice?,"""Testing that dynamically responds to the behavior of the test object and to test results being obtained"" (Hamburg and Mogyorodi, 2024)","Dynamic Testing (implied by Hamburg and Mogyorodi, 2024)",,"What does ""responds"" mean in this context?"
Recoverability Testing,"Type (implied by its quality (ISO/IEC, 2023a; IEEE, 2017, p. 370; Washizaki, 2024, p. 5-9))","Testing ""how well a system or software can recover data during an interruption or failure"" (Washizaki, 2025, p. 7-10; similar in IEEE, 2024, p. 17; 2017, p. 369; OG ISO/IEC, 2011) and ""re-establish the desired state of the system"" (IEEE, 2024, p. 17; 2017, p. 369; OG ISO/IEC, 2011)","Reliability Testing (IEEE, 2017, p. 375; Washizaki, 2025, p. 7-10), Usability Testing? (p. 5-10), Dependability Testing (if it exists) (IEEE, 2024, pp. 14, 21, 23, 62-66; 2017, p. 375)","Resilience Testing (IEEE, 2024, p. 17), Recovery Testing (Kam, 2008, p. 47)","Related to failover testing (Washizaki, 2024, p. 5-9) and survivability testing (IEEE, 2017, p. 450). Can be tested using injection slots (p. 225) and improved through redundancy (p. 370)"
Recovery Testing,"Type (IEEE, 2022, p. 22)","""Testing … aimed at verifying software restart capabilities after a system crash or other disaster"" (Washizaki, 2025, p. 5-9) including ""recover[ing] the data directly affected and re-establish[ing] the desired state of the system"" (ISO/IEC, 2023a) so that the system ""can perform required functions"" (IEEE, 2017, p. 370)","Performance-related Testing (IEEE, 2022, p. 22), Reliability Testing (IEEE, 2021c, Tab. A.1; Washizaki, 2024, pp. 4-11, 7-10), Availability Testing (IEEE, 2017, p. 38), Non-functional Testing (Washizaki, 2025, p. 5-9), Fault Tolerance Testing (2024, p. 4-11), Continuity Testing (implied by Washizaki, 2025, p. 6-8; Koomen et al., 2006, p. 50), Dynamic Testing? (implied by Gerrard, 2000a, Fig. 5)","Recoverability Testing (Kam, 2008, p. 47), Restore Testing (implied by Washizaki, 2025, p. 6-8; Koomen et al., 2006, p. 50), Restart & Recovery (Testing)? (Gerrard, 2000a, Fig. 5)","May be done through simulations (Washizaki, 2024, p. 5-28). Difference between this and disaster/recovery testing?"
"Red Team Testing (Firesmith, 2015, p. 57)","Type (implied by Firesmith, 2015, p. 57)",,"Penetration Testing (Firesmith, 2015, p. 57)",,
Regression Testing,"Type (Hamburg and Mogyorodi, 2024; Koomen et al., 2006, pp. 9, 51), Level (Lyu, 1996, p. 532; Barbosa et al., 2006, p. 3)","Testing ""performed following modifications to a test item … or to its operational environment"" (IEEE, 2022, p. 8; 2021a, p. 3; similar in Kam, 2008, p. 47), such as refactoring (Lahiri et al., 2013, p. 345) or bug fixes (p. 346; Patton, 2006, p. 232), to ensure they have ""not introduced additional defects"" (ISO/IEC, 2014c; similar in IEEE, 2021a, p. 3; Koomen et al., 2006, p. 51; Patton, 2006, p. 232) or ""unwanted effects"" (Peters and Pedrycz, 2000, p. 446; similar in IEEE, 2022, p. 8; 2021a, p. 3), primarily ""in unmodified parts of the test item"" (2022, p. 8; 2021a, p. 3; similar in Hamburg and Mogyorodi, 2024; Koomen et al., 2006, p. 51)","Functional Testing, Non-functional Testing, Continuous Testing, Agile Testing, DevOps Testing (Washizaki, 2024, p. 5-8), Change-Related Testing (Hamburg and Mogyorodi, 2024), Web Application Testing (Doğan et al., 2014, Tabs. 8, 22, p. 185), System Testing (can be) (IEEE, 2021b, p. 45), Developer Testing (can be) (Gerrard, 2000a, p. 11), Automated Testing (should be) (Peters and Pedrycz, 2000, p. 481; implied by Patton, 2006, p. 231-232), Retesting (incorrectly) (IEEE, 2017, p. 372; Washizaki, 2024, pp. 5-8, 6-5, 7-5 to 7-6), Relative Correctness Testing","Non-regression Testing (Washizaki, 2025, p. 5-8; implied by Koomen et al., 2006, p. 51)","Can be done with a ""retest-all"" approach where ""all tests are rerun"" or a ""selective retest"" approach where ""only some of the tests are rerun"", including those ""in which the modified and original program produce different results"" (van Vliet, 2000, p. 411); are these separate approaches? Should be part of any test strategy and its level should be ""based on a knowledge of the risks associated with developers making changes"" (IEEE, 2022, p. 23); it can be applied to any level but ""is usually performed after integration testing"" (Washizaki, 2024, p. 5-8). ""Play[s] an important role in software engineering operations"" (p. 6-5) and ""maintenance"" (p. 7-5) by focusing on ""the re-release of a modified software product"" (Lyu, 1996, p. 532). Its effectiveness is expressed in terms of the difficulty of test suite construction and maintenance and the reliability of the testing system (Peters and Pedrycz, 2000, pp. 481-482). It is sometimes desired, although tedious, to allow ""some unwanted differences to pass through"" (p. 446). Can be supported by tools, such as ""tinderboxes"" (IEEE, 2017, p. 478). ISO/IEC and IEEE make a distinction between ""regression testing"" and a ""regression test"" (IEEE, 2017, p. 372)"
Regulatory Acceptance Testing,"Level (Firesmith, 2015, p. 30; inferred from acceptance testing)","""Testing performed to determine the compliance of the test object"" (Hamburg and Mogyorodi, 2024)","Acceptance Testing (Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 30)","Regulation Acceptance Testing (RAT)? (Firesmith, 2015, p. 30)",Difference between this and acceptance testing?
"Relative Correctness Testing (inferred from relative correctness (Lahiri et al., 2013, p. 345) and correctness testing)",Type (inferred from correctness testing),,Correctness Testing,,"Weaker than absolute correctness testing (Lahiri et al., 2013, p. 345)"
"Reliability Enhancement Testing (Firesmith, 2015, p. 55)","Type (implied by Firesmith, 2015, p. 55)",,"Reliability Testing (Firesmith, 2015, p. 55)",,
"Reliability Growth Testing (Firesmith, 2015, p. 55)","Type (implied by Firesmith, 2015, p. 55)",,"Reliability Testing (Firesmith, 2015, p. 55)",,
"Reliability Mechanism Testing (Firesmith, 2015, p. 55)","Type (implied by Firesmith, 2015, p. 55)",,"Reliability Testing (Firesmith, 2015, p. 55)",,
Reliability Testing,"Type (IEEE, 2022, pp. 8, 22; 2021c, p. 40, Tab. A.1; 2017, p. 375; 2016, p. 4; 2013, p. 5; implied by its quality (ISO/IEC, 2023a; IEEE, 2017, p. 361; 2015, p. 8; Bertolino et al., 2019, p. 93:30); Firesmith, 2015, p. 53)","Testing that evaluates the ""capability of a product to perform specified functions under specified conditions for a specified period of time without interruptions and failures"" (ISO/IEC, 2023a; similar in IEEE, 2024, pp. 17-18; 2021c, p. 40; Lyu, 1996, pp. 534, 539); can be measured as ""the frequency with which failures occur"" (2022, p. 8; 2021c, p. 40; 2013, p. 5) or the ""probability that software will not cause the failure of a system"" (IEEE, 2017, p. 425)","Model-based Testing, Operational Profile Testing (IEEE, 2021c, p. 40), Functionality Testing (2017, p. 375), Non-functional Testing (Washizaki, 2025, p. 5-9; Bertolino et al., 2019, pp. 93:11, 93:19; Kam, 2008, pp. 46-47; Gerrard, 2000a, Tab. 2; 2000b, Tab. 1, p. 22), Web Application Testing (Doğan et al., 2014, p. 185), Performance Testing (Gerrard, 2000b, pp. 23, 26), Automated Testing (2000a, Tab. 2; 2000b, Tab. 1, p. 26), Smoke Testing, Dynamic Testing (2000a, Tab. 2; 2000b, Tab. 1), Regression Testing (can be) (ISO/IEC, 2014c; Washizaki, 2025, p. 5-8), Security Testing (implied by IEEE, 2017, p. 404), Time-domain based Testing (can be), Structure-based Testing (can be) (Lyu, 1996, p. 539), Dependability Testing (if it exists) (IEEE, 2024, pp. 14, 21, 23, 36-48; 2017, p. 375; ISO/IEC, 2023a)","Soak Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1, p. 26)","Often uses statistical models of user behaviour, reliability growth models (Washizaki, 2024, p. 5-9), and/or random testing (van Vliet, 2000, p. 439) and may be facilitated by DevOps (Washizaki, 2024, p. 5-9) such as ""infrastructure/operations services"", which can be done early on in the development process (Washizaki, 2024, p. 6-5). Can be done using ""the test-retest method, the alternative form method, the split-halves method and the internal consistency method"" (verify this applies to software) (Washizaki, 2024, p. 18-14) and can find ""obscure bugs"" ""over extended periods"" (Gerrard, 2000b, p. 23), which is how they should be performed (p. 26)"
"Remote Testing (Jard et al., 1999)",Practice?,,"Asynchronous Testing (implied by Jard et al., 1999), Manual Testing (implied by p. 26)",,"""Error-prone"" (Jard et al., 1999, p. 25)"
Repetition Testing,Type? Practice?,"Testing ""the same operation over and over"", potentially up to ""thousands of attempts"" (Patton, 2006, p. 86)","Negative Testing (Patton, 2006, p. 86)",,
"Request Testing (Doğan et al., 2014, Tab. 13; OG Peng and Lu, 2011)",Technique (inferred from coverage-based testing),,"Coverage-based Testing, Web Application Testing (Doğan et al., 2014, Tab. 13)",,
Requirement(s)-based Testing,"Technique (IEEE, 2022, pp. 8, 22; 2021c, pp. 5, 22-23, Fig. 2; implied by Firesmith, 2015, p. 47)","Testing ""based on test objectives and test conditions derived from [atomic (IEEE, 2022, p. 8; 2021c, pp. 5, 22), often textual (p. 22)] requirements, e.g. tests that exercise specific functions or probe non-functional attributes such as reliability or usability"" (Kam, 2008, p. 47)","Specification-based Testing (IEEE, 2022, pp. 8, 22; 2021c, pp. 5, 22, Fig. 2; implied by Firesmith, 2015, p. 47), Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, p. 3; 2021c, pp. 6, 22), Risk-based Testing (2022, p. 20), User-session-based Testing (Doğan et al., 2014, p. 183; implied by p. 193), Positive Testing (typically) (IEEE, 2021c, p. 23), Conformance Testing (implied by Washizaki, 2024, p. 5-7)","Software Quality Evaluations? (ISO/IEC, 2014b, p. 6), Requirements Testing? (Firesmith, 2015, p. 47)","May involve ""requirements analysis"". In general, each requirement should be unambiguous, testable, binding, and “acceptable to all stakeholders”, and the “overall collection” should be complete, consistent, and feasible (Washizaki, 2024, p. 1-8). See Pinkster et al., 2006"
Requirements Animation,Approach,,"Static Testing, W-Model Testing (Gerrard, 2000a, Fig. 4), Requirements-driven Testing",,
"Requirements Engineer Testing (Firesmith, 2015, p. 39)",Practice?,,"Developer Testing (Firesmith, 2015, p. 39), Requirements-driven Testing?",,
Requirements-driven Testing,Approach,"Testing that ""verifies"" the ""system meets system requirements"" (Firesmith, 2015, p. 33)",,,Difference between this and requirements-based testing?
"Resource Utilization Testing (Kam, 2008, p. 47)","Type (implied by its quality (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1))","The ""capability of a product to use no more than the specified amount of resources to perform its function under specified conditions"" (ISO/IEC, 2023a)","Performance Efficiency Testing (ISO/IEC, 2023a; IEEE, 2021c, Tab. A.1), Performance Testing (Peters and Pedrycz, 2000, p. 447), Efficiency Testing (implied by Kam, 2008, p. 47)","Storage Testing? (but this seems incomplete) (Hamburg and Mogyorodi, 2024)",
Response-Time Testing,"Type (implied by its quality (ISO/IEC, 2023a))","Testing that evaluates the ""capability of a product to perform its specified function under specified conditions so that the response time … meet[s] the requirements"" (ISO/IEC, 2023a)","Requirements-based Testing (ISO/IEC, 2023a), Performance-related Testing (IEEE, 2022, p. 22), Performance Testing (Washizaki, 2025, p. 5-9; Peters and Pedrycz, 2000, p. 447; implied by Hamburg and Mogyorodi, 2024), Timing Testing? (implied by Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), (Time Behaviour Testing (ISO/IEC, 2023a))",,"Good to use if the software may have ""a large number of concurrent users"" (IEEE, 2022, p. 45). Captured as part of ""object load and timing"" by Gerrard (2000a, Tab. 2; 2000b, Tab. 1)"
Retesting,"Type (Hamburg and Mogyorodi, 2024)","Testing ""performed to check that modifications made to correct a fault have successfully removed the fault"" (IEEE, 2022, p. 8; 2021a, p. 3; similar in 2017, p. 386; Hamburg and Mogyorodi, 2024), often by rerunning test cases that previously failed (IEEE, 2022, p. 35; 2017, p. 386; Kam, 2008, p. 47) that may be ""supplemented by new test cases that provide improved coverage"" (IEEE, 2022, p. 35)","Change-Related Testing (Hamburg and Mogyorodi, 2024)","Sometimes spelled with a hyphen (Hamburg and Mogyorodi, 2024), Confirmation Testing (IEEE, 2022, pp. 8, 35; 2021a, p. 3; 2017, p. 386; Hamburg and Mogyorodi, 2024)","Should be part of any test strategy and its level should be ""based on a knowledge of the risks associated with developers making changes"" (IEEE, 2022, p. 23). Often complemented by regression testing (p. 8; 2021a, p. 3)"
"Reuse Testing (Firesmith, 2015, p. 34)",Approach,,,,
Reviews,Approach,"""Process[es] or meeting[s] during which a software product is presented to project personnel, managers, users, customers, user representatives, or other interested parties for comment or approval"" (IEEE, 2017, p. 388) ""to detect defects or to provide improvements"" (Hamburg and Mogyorodi, 2024), such as adherence to standards and guidelines (see Patton, 2006, pp. 96-103)","Static Testing (IEEE, 2022, pp. 9, 17, 25, 28; 2021a, pp. 3, 21; Hamburg and Mogyorodi, 2024; Gerrard, 2000a, p. 12, Fig. 4; 2000b, p. 3), W-Model Testing (Gerrard, 2000a, Fig. 4), Structural Analysis, Structured Testing, Role-based Testing (Patton, 2006, p. 92), Static Analysis? (Washizaki, 2024, p. 12-13), Reliability Testing (can be), Maintainability Testing (can be), Portability Testing (can be), (Readability Testing) (Patton, 2006, p. 96)","Structural Analysis (implied by Patton, 2006, p. 92)","""Can identify issues early in development or even before a component is designed"" (Washizaki, 2024, p. 12-13; implied by Gerrard, 2000a, pp. 9-10) and improve team performance and code quality, even before the review: ""if a developer knows that his work is being carefully reviewed by his peers, he might make an extra effort to … make sure that it's right"" (Patton, 2006, pp. 93-94). Consist of four main parts: identifying problems, following rules (i.e., structure), preparing in advance, and summarizing results afterwards (p. 93). See ISO/IEC 20246; van Vliet, 2000, pp. 415-417; Peters and Pedrycz, 2000, pp. 482-485"
ReWeb Testing,Approach,,"Web Application Testing, UML Testing, Structure-based Testing, Control Flow Testing, Functional Testing (Kam, 2008, Tab. 1)",,
Risk-based Testing,"Technique (Firesmith, 2015, p. 47; implied by IEEE, 2022, p. 18; 2021c, p. vii)","Testing ""in which the management, selection, prioritization, and use of testing activities and resources are consciously based on corresponding types and levels of analysed risk"" (IEEE, 2022, p. 8; 2021a, p. 3; 2017, p. 394; OG 2013)","Sampling (IEEE, 2022, p. 18), Specification-based Testing (Firesmith, 2015, p. 47)",,"Recommended technique for sampling (IEEE, 2022, p. 18) and helps to ""determine the set of techniques … applicable in specific situations"" (2021c, p. vii) ""for a given project or organization"" (p. 7) and ""design[] effective tests to find faults"" without ""worry[ing] about doing 'too little' testing"" (Gerrard, 2000a). May involve ""risk analysis"". See the ISO/IEC 25000 SQuaRE family of standards and Gerrard"
Robustness Testing,"Type (implied by Firesmith, 2015, p. 53)","Testing the ""degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions"" (IEEE, 2017, p. 394)","Boundary-Value Analysis (Washizaki, 2024, p. 5-11), Non-functional Testing (Bertolino et al., 2019, p. 93:11), Stress Testing (implied by Moghadam, 2019, p. 1188), Survivability Testing (Ghosh and Voas, 1999, p. 42)","Fault Tolerance Testing (Hamburg and Mogyorodi, 2024)","Related to error/fault tolerance (IEEE, 2017, p. 394) and potentially stress testing; may be supported by fault injection (Ghosh and Voas, 1999, pp. 42-44)"
Role-based Reviews,Approach,"Reviews ""in which a work product is evaluated from the perspective of different stakeholders"" (Hamburg and Mogyorodi, 2024)","Reviews (Hamburg and Mogyorodi, 2024), Role-based Testing, Scenario-based Testing?",,
"Role-based Testing (Firesmith, 2015, p. 39)",Practice?,,,,
Runtime Assertion Checking (RAC),Practice?,"""Testing … [separation logic] specifications during program execution"", with ""any violations result[ing] in special errors being reported"" (Chalin et al., 2006, p. 343)","Assertion Checking (Chalin et al., 2006, p. 343), Dynamic Analysis?, Specification-based Testing?",,
Safety Demonstrations,"Artifact (implied by IEEE, 2017, p. 397), Approach","""Bod[ies] of evidence and rationale that show[] an item is justified as being safe within allowed limits on risk"" (IEEE, 2017, p. 397; OG ISO/IEC, 2011)",Safety Testing,,
"Safety Engineer Testing (Firesmith, 2015, p. 39)",Practice?,,"Developer Testing (Firesmith, 2015, p. 39), Safety Testing",,
"Safety Testing (Firesmith, 2015, p. 53; Kam, 2008, p. 47)","Type (implied by its quality (ISO/IEC, 2023a; IEEE, 2017, pp. 361, 397, 427; OG 1994; 2015, p. 8); Firesmith, 2015, p. 53)","Testing the ""capability of a product under defined conditions to avoid a state in which human life, health, property, or the environment is endangered"" (ISO/IEC, 2023a; similar in IEEE, 2024, p. 18; 2017, p. 397) or ""freedom from software hazards"" (IEEE, 2017, p. 427; OG 1994)",,,"May be based on safety requirements (see IEEE, 2017, p. 398; OG ISO/IEC, 2011) and/or supported by fault injection (Ghosh and Voas, 1999, p. 39). See ISO/IEC/IEEE 12207:2017 and ISO/IEC Guide 51"
Sampling,Technique,"Testing that is derived from the ""set of possible input combinations and states"" by choosing subsets ""that are most likely to uncover issues of interest"" (IEEE, 2022, p. 18)",,,"In contrast to exhaustive testing (IEEE, 2022, p. 18)"
Sandwich Testing,"Technique (Sharma et al., 2021, pp. 601, 603, 605-606), Level (inferred from integration testing)","Testing that ""combines the ideas of bottom-up and top-down testing"" by working towards a ""target layer"" from either end of the hierarchy using the relevant test approach (Peters and Pedrycz, 2000, Tab. 12.8); allows for ""slices"" of the system to be tested across multiple layers (inferred from Sangwan and LaPlante, 2006, Fig. 3)","Integration Testing (Washizaki, 2024, p. 5-7; Sharma et al., 2021, p. 603; Sangwan and LaPlante, 2006, p. 27; Peters and Pedrycz, 2000, p. 488, Tab. 12.8)","Mixed Testing (Washizaki, 2024, p. 5-7)","Often more useful than its parts in practice (van Vliet, 2000, p. 439)"
Scalability Testing,"Type (implied by its quality (ISO/IEC, 2023a; Bertolino et al., 2019, p. 93:30); Firesmith, 2015, p. 53)","Testing the capability of a product to ""perform under conditions that may need to be supported in the future"" (IEEE, 2021c, p. 39; similar in Hamburg and Mogyorodi, 2024; Gerrard and Thompson, 2002), including ""what level of additional resources (e.g. memory, disk capacity, network bandwidth) will be required to support anticipated future loads"" (IEEE, 2021c, p. 39; similar in Washizaki, 2025, pp. 5-5, 5-9), or ""to handle growing or shrinking workloads or to adapt its capacity to handle variability"" (ISO/IEC, 2023a)","Portability Testing (ISO/IEC, 2023a), Load Testing, Volume Testing, Requirements-based Testing, Transaction Flow Testing? (Washizaki, 2025, pp. 5-5, 5-9), Non-functional Testing (pp. 5-8 to 5-9), Elasticity Testing, Stress Testing (p. 5-9), Performance Testing, Efficiency Testing, Performance Efficiency Testing? (Pandey, 2023), Backup Testing (can be) (Bas, 2024, pp. 22-23), Cloud Testing (can be) (Washizaki, 2025, p. 5-27), Memory Management Testing? (implied by IEEE, 2021c, p. 39), Modifiability Testing (if it exists)","Capacity Testing (IEEE, 2021c, p. 39)","""Connected to the complexity of the platform and environment in which the program runs"" (Washizaki, 2025, p. 5-5) and is ""particularly important in distributed or high-performance systems"" (2024, p. 5-9). May be performed ""us[ing] infrastructure/operations services"" early on in the development process (p. 6-5)"
Scenario Testing,"Technique (IEEE, 2022, pp. 9, 22; 2021c, pp. 5, 8, 20, Fig. 2; 2017, p. 400; OG 2013; Washizaki, 2024, p. 5-12; Firesmith, 2015, p. 47; Sangwan and LaPlante, 2006, p. 26)","Testing ""based on exercising sequences of interactions between the test item and other systems,"" including users (IEEE, 2022, p. 9; 2021c, p. 5; similar on p. 20), to test ""usage flows involving the test item""; this should include ""typical"" and ""alternative"" scenarios, with the latter including ""abnormal use, extreme or stress conditions, exceptions and error handling"" (p. 20)","Specification-based Testing (IEEE, 2022, pp. 9, 22; 2021c, pp. 5, 20, Fig. 2; Washizaki, 2024, p. 5-12; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 47), Model-based Testing (IEEE, 2021c, p. 20; implied by Bourque and Fairley, 2014, p. 4-10), System Testing, Acceptance Testing (can be) (IEEE, 2021c, p. 20), Stress Testing, Error-based Testing, End-to-end Testing, Functional Testing (implied by p. 20), Object-Oriented Testing (implied by Sangwan and LaPlante, 2006, p. 27)","Use Case Testing, User Scenario Testing (Hamburg and Mogyorodi, 2024)","""Often used with test automation harnesses"" (Washizaki, 2024, p. 5-12). See Desikan and Ramesh 2007"
Scenario Walkthroughs,Approach,,"Static Testing, W-Model Testing (Gerrard, 2000a, Fig. 4), Walkthroughs, Scenario Testing",,
"Scenario-based Evaluations (van Vliet, 2000, pp. 417-418)",Technique,,"Structural Analysis? (van Vliet, 2000, pp. 417-418), Scenario-based Testing?",,
"Scenario-based Testing (Sangwan and LaPlante, 2006, p. 26)",Approach,,,"(Use) Scenario Testing? (Sangwan and LaPlante, 2006, p. 26)",
Scenario-based Reviews,Approach,"Testing ""to determine [a work product's] ability to address specific scenarios"" (Hamburg and Mogyorodi, 2024)","Reviews (Hamburg and Mogyorodi, 2024), Scenario-based Testing",,
Scripted Testing,"Practice (IEEE, 2022, pp. 20, 22; implied by p. 33)","Testing ""performed based on a documented test script"" (IEEE, 2022, p. 9; 2021a, p. 3; 2017, p. 403; OG 2013) or ""written instructions in a test case"" (IEEE, 2017, p. 403; OG 2013)","Dynamic Testing (IEEE, 2017, p. 403; OG 2013), Manual Testing (usually) (IEEE, 2022, pp. 9, 33; 2021a, p. 3; 2017, p. 403; Hamburg and Mogyorodi, 2024), Formal Testing (usually) (IEEE, 2022, p. 33), Automatic Testing (""typically requires more detail"") (p. 33; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 44)","Test Scripting? (Hamburg and Mogyorodi, 2024), Script-based Testing? (Firesmith, 2015, p. 44)","""Helps achieve required test coverage levels"" (IEEE, 2022, p. 33). ""Automated test scripts may either be generated automatically by a framework or developed manually by a test automation specialist"" (2016, p. 7)"
"Search-based Testing (Engström and Petersen, 2015, pp. 1-2)","Technique (Engström and Petersen, 2015, pp. 1-2)",,,"Sometimes spelled without a hyphen (Engström and Petersen, 2015, pp. 1-2)",
Security Attacks,Approach,"""Attempt[s] to gain unauthorized access to a component or system, resources, information, or … to compromise system integrity"" (Hamburg and Mogyorodi, 2024)","Security Testing, Attacks, Integrity Testing (Hamburg and Mogyorodi, 2024)",,"""An example of a security attack would be to block access to software libraries"" (IEEE, 2022, p. 34). Could be malicious or as a genuine part of testing; see note on attacks. See NIST.IR.7298"
Security Audits,"Technique (IEEE, 2021c, p. 40)","Testing ""to determine whether any security vulnerabilities are present"" (IEEE, 2021c, p. 40) or with the ""aim to ensure that all of the products installed on a site are secure when checked against the known vulnerabilities for those products"" (Gerrard, 2000b, p. 28)","Security Testing, Audits, Static Testing, Inspections, Reviews, Walkthroughs (IEEE, 2021c, p. 40), Manual Testing (can be, often done by consultants) (Gerrard, 2000b, p. 34), Automated Testing (can be, through scans ""that perform ping sweeps, port scans, [and] operating system detection"") (p. 28)",,
"Security Engineer Testing (Firesmith, 2015, p. 39)",Practice?,,"Developer Testing (Firesmith, 2015, p. 39), Security Testing",,
Security Testing,"Type (IEEE, 2022, pp. 9, 22, 26-27; 2021a, pp. 3, 8, 11; 2021b, Fig. 2; 2021c, pp. 7, 40, Tab. A.1; 2017, p. 405; OG 2013; Koomen et al., 2006, p. 9; implied by its quality (ISO/IEC, 2023a; IEEE, 2017, p. 361; 2015, p. 8; Washizaki, 2024, p. 13-4; Bertolino et al., 2019, p. 93:30); Firesmith, 2015, p. 53)","Testing that evaluates the ""capability of a product to protect information and data [including data in transmission (ISO/IEC, 2023a)] so that persons or other products have the degree of data access appropriate to their types and levels of authorization, and to defend against attack patterns by malicious actors"" (ISO/IEC, 2023a; similar in IEEE, 2022, p. 9; 2021a, p. 3; 2021c, p. 40; Washizaki, 2024, pp. 5-9, 13-4)","Reliability Testing (ISO/IEC, 2023a), Dynamic Testing (IEEE, 2021a, pp. 11, 29; Gerrard, 2000a, Fig. 5, Tab. 2), Model-based Testing, Requirements-based Testing, Conformance Testing (IEEE, 2021c, p. 40), Non-functional Testing (Bertolino et al., 2019, p. 93:11, Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Functionality Testing (2000a, Tab. 2; 2000b, Tab. 1; implied by Kam, 2008, p. 47), Sys Admin Testing (Gerrard, 2000b, p. 29), Post-Deployment Monitoring (pp. 32-33), Experience-based Testing (can be) (IEEE, 2022, p. 4; 2021c, p. 4), Automated Testing (can be) (2021b, p. 10), Backup Testing (can be) (Bas, 2024, pp. 14, 18, Tab. 3), Cloud Testing (can be) (Washizaki, 2025, p. 5-27), Compliance Testing (implied by Bas, 2024, pp. 26-27), System Testing? (implied by Gerrard, 2000a, Fig. 5), Dependability Testing (if it exists) (ISO/IEC, 2023a; IEEE, 2017, p. 375)","Information Security Testing (Hamburg and Mogyorodi, 2024)","Good to use when the software will be widely available (IEEE, 2022, p. 45) and may be supported by fault injection (Ghosh and Voas, 1999, p. 39)"
"Self-Testing (Firesmith, 2015, p. 31)",Approach,,"Ongoing Built-In Testing, Periodic Built-In Testing (Firesmith, 2015, p. 31)",Built-In Testing?,
Session-based Testing,"Approach (Hamburg and Mogyorodi, 2024)","Testing ""in which test activities are planned as test sessions"" (Hamburg and Mogyorodi, 2024), potentially based on navigation models or logs (Doğan et al., 2014, Tab. 22)","Web Application Testing (Doğan et al., 2014, Tab. 22)",,
"Shift-Left Testing (Washizaki, 2024, p. 9-5)",Practice?,,Lifecycle-based Testing?,,
Shoe Testing,"Technique (Firesmith, 2015, p. 51)",,"Random Testing (Firesmith, 2015, p. 51)",,
"Shutdown Built-In Testing (Firesmith, 2015, p. 31)",Practice?,,"Built-In Testing (Firesmith, 2015, p. 31)","SBIT (Firesmith, 2015, p. 31)",
Sign Change Testing,Approach,"Testing of ""the coverage of neurons activated with both positive and negative activation values in a neural network for a set of tests"" (Hamburg and Mogyorodi, 2024)",ML Model Testing,,
Sign-Sign Testing,Approach,"Testing of ""the coverage achieved if by changing the sign of each neuron it can be shown to individually cause one neuron in the next layer to change sign while all other neurons in the next layer do not change sign for a set of tests"" (Hamburg and Mogyorodi, 2024)",ML Model Testing,,See ISO 29119-11
Similarity-based Prioritization Testing,Practice?,"Testing that prioritizes test cases ""starting from those most dissimilar according to a predefined distance function"" (Washizaki, 2024, p. 5-8)","Prioritization Testing (Washizaki, 2024, p. 5-8)",,
Single-Hit Decision Table,Technique (inferred from decision table testing),"Testing ""based on exercising decision rules"" (IEEE, 2022, p. 4) in a single-hit decision table (IEEE, 2017, p. 415; OG ISO, 1984)","Decision Table Testing (IEEE, 2017, p. 175; OG ISO, 1984)",,
Site Acceptance Testing,"Level (Firesmith, 2015, p. 30; inferred from acceptance testing)","""Acceptance testing by users/customers at their site, [sic] to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes"" (Kam, 2008, p. 47)","Acceptance Testing (Firesmith, 2015, p. 30; Kam, 2008, p. 47)","SAT (Firesmith, 2015, p. 30)","""Normally includ[es] hardware as well as software"" (Kam, 2008, p. 47)"
"Slicing (Testing) (Kam, 2008, Tab. 1, pp. 26-29)",Level?,"Testing a subset of a program by reducing its size while preserving its original behaviour (Kam, 2008, p. 26)","Model-based Testing (Kam, 2008, Tabs. 1-2), Web Application Testing (can be) (p. 26), Data Flow Testing (implied by p. 26; OG Ricca et al [32, 35])",,
Smoke Testing,"Technique (Washizaki, 2025, p. 5-14; Sharma et al., 2021, pp. 601, 603, 605-606)","Testing that ""ensures that the SUT's core functionalities behave properly"" (Washizaki, 2025, p. 5-14; similar in Kam, 2008, p. 47) and that ""the SUT is operational before the planned testing begins"" (Washizaki, 2025, p. 5-14; similar in Hamburg and Mogyorodi, 2024)","Ad Hoc Testing, Quick Testing (Washizaki, 2025, p. 5-14), Unscripted Testing (implied by p. 5-14; Hamburg and Mogyorodi, 2024), Integration Testing (Sharma et al., 2021, p. 603), Automated Testing (often) (Gerrard, 2000a, p. 13), Build Verification Testing (implied by Hamburg and Mogyorodi, 2024), Offline Testing?","Build Verification Testing (Washizaki, 2025, p. 5-14), Sanity Testing, Intake Testing, Confidence Testing (Hamburg and Mogyorodi, 2024)","Should be prioritized in the presence of ""severe time pressure"" (Gerrard, 2000a, p. 13) and ""prevents failures because of the test environment"" (Washizaki, 2025, p. 5-14). Seems to be implied to be the responsibility of DevOps (Washizaki, 2024, p. 6-9)"
"SOA Testing (Doğan et al., 2014, Tab. 1)",Approach,,,,"See Palacios et al., (2011) and Mesbah and Prasad (2011)"
Software Design Audits,Practice (inferred from audits),"""Review[s] of … software product[s] to determine compliance with requirements, standards, and contractual documents"" (IEEE, 2017, p. 420)","Audits, Reviews (IEEE, 2017, p. 420)",,"Is this specific to the software's design, or to the software itself?"
"Software Interaction Testing (Valcheva, 2013, p. 468)",Approach,,,,
Software Qualification Testing,Level?,"""Testing performed on completed, integrated software to provide evidence for compliance with software requirements"" (Hamburg and Mogyorodi, 2024)","Safety Testing (implied by Knüvener Mackert GmbH, 2022, pp. 36-37), System Testing?, Dynamic Testing?",,Difference between this and system testing?
"Software-in-the-Loop Testing (Firesmith, 2015, p. 23)",Level (inferred from system testing),"""Testing performed using real software in a simulated environment or with experimental hardware"" (Hamburg and Mogyorodi, 2024)","System Testing (Firesmith, 2015, p. 23), Formal Testing (Preuße et al., 2012, p. 3), Dynamic Testing (Hamburg and Mogyorodi, 2024), Integration Testing? (Knüvener Mackert GmbH, 2022, p. 153)","SiL Testing (Hamburg and Mogyorodi, 2024; Preuße et al., 2012, p. 3), SIL Testing (Firesmith, 2015, p. 23), SiL Verification? (Preuße et al., 2012, p. 7)","Used by ""the scientific community"" and is ""vendor- and hardware-independent"" (Preuße et al., 2012, p. 2)"
"SoS Integration Testing (Firesmith, 2015, p. 23)",Level (inferred from system testing),"Testing that ensures the ""synthesi[s of] a set of system elements into a realized system (product or service) … satisfies system requirements, architecture, and design"" (IEEE, 2019b, p. 48)","SoS Testing, Operational Testing (sometimes) (IEEE, 2019b, pp. 48-49), System Testing (Firesmith, 2015, p. 23), Integration Testing",,"""Could be restricted to modeling or analysis"" (IEEE, 2019b, p. 49)"
SoS Testing,Level (inferred from system testing),"The testing of a system of systems (IEEE, 2019b)","System Testing (Firesmith, 2015, p. 23)","System(s) of Systems Testing (IEEE, 2019b, p. 3; 2015, p. 11)","""Because SoS can contain system elements in addition to CS, resources and budgets to support those elements could be needed"" (IEEE, 2019b, p. 19). See ISO/IEC/IEEE 21839"
Specification-based Testing,"Technique (IEEE, 2022, p. 22; 2021b, p. 45; 2021c, p. 8; Washizaki, 2025, p. 5-10; Hamburg and Mogyorodi, 2024; Firesmith, 2015, pp. 46-47; Souza et al., 2017, p. 3; Sakamoto et al., 2013, p. 344; implied by IEEE, 2022, pp. 2-4, 6-9), Type (Hamburg and Mogyorodi, 2024)","""Testing in which the principal test basis is the external inputs and outputs of the test item"" (IEEE, 2022, p. 9; 2017, p. 431) or its ""requirements, specifications, models [and/]or user needs"" (2021c, p. 8) that ""select[s] a few test cases from the input domain that can detect specific categories of faults"" (Washizaki, 2025, p. 5-10)","Dynamic Testing (IEEE, 2022, p. 17; Sharma et al., 2021, Fig. 1), Model-based Testing? (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, p. 3; 2021c, p. 6), Conformance Testing (implied by Washizaki, 2024, p. 5-7)","Black-Box Testing (IEEE, 2022, p. 9; 2021c, p. 8; 2017, p. 431; Washizaki, 2025, p. 5-10; Hamburg and Mogyorodi, 2024; Firesmith, 2015, pp. 46-47 (without hyphen); van Vliet, 2000, p. 399; Sakamoto et al., 2013, p. 344), Closed-Box Testing (IEEE, 2022, p. 9; 2017, p. 431), Domain Testing (Washizaki, 2025, p. 5-10), Specification-oriented Testing (Peters and Pedrycz, 2000, p. 440, Fig. 12.2), Input Domain-based Testing (implied by Bourque and Fairley, 2014, pp. 4-7 to 4-8)","Should be done before structure-based testing to avoid bias towards ""test cases based on how the module works"" (Patton, 2006, p. 113). ""Effective at detecting errors of omission"" (IEEE, 2021c, p. 128). Difference between this and functional testing (van Vliet, 2000, p. 399; IEEE, 2017, p. 431; Souza et al., 2017, p. 3)? Umar (2020) has more synonyms"
Spike Testing,Approach,"""Testing to determine the ability of a system to recover from sudden bursts of peak loads and return to a steady state"" (Hamburg and Mogyorodi, 2024)",,,
Spiral Testing,Practice?,"Testing ""performed iteratively until the software is complete"" alongside other phases of development (IEEE, 2017, p. 432)","Lifecycle-based Testing (Washizaki, 2024, p. 10-5; OG [2, 3, 10]), Risk-based Testing? (Washizaki, 2024, p. 10-6; OG [3]), Continuous Testing",,
"Stability Testing (Firesmith, 2015, p. 55)","Type (implied by its quality (IEEE, 2017, p. 434; OG ISO/IEC, 2009) and Firesmith, 2015, p. 55)","Testing a ""property that an object has with respect to a given failure mode if it cannot exhibit that failure mode"" (IEEE, 2017, p. 434; OG ISO/IEC, 2009)","Build Verification Testing (Hamburg and Mogyorodi, 2024), Reliability Testing? (Firesmith, 2015, p. 55), Modifiability Testing (if it exists) (ISO/IEC, 2023a)",,"Example: endurance stability testing (Firesmith, 2015, p. 55)"
State Testing,"Technique (inferred from state transition testing and IEEE, 2021c, p. 19)","Testing ""the program's logic flow through its various states"" (Patton, 2006, p. 79) by verifying the values of state variables (p. 83) in which ""all states in the state model … [are] 'visited'"" (IEEE, 2021c, p. 19) and/or the most/least common transitions, error states/transitions, and/or random state transitions are tested (Patton, 2006, p. 82-83)","Model-based Testing (IEEE, 2021c, p. 19; Doğan et al., 2014, p. 184), State Transition Testing (IEEE, 2021c, p. 19), Behavioural Testing (Patton, 2006, p. 79-87)","All States Testing (implied by IEEE, 2021c, p. 19; Kam, 2008, p. 15)",
State Transition Testing,"Technique (IEEE, 2022, pp. 9, 22; 2021c, pp. 5, 19, Fig. 2; Washizaki, 2024, p. 5-12; Hamburg and Mogyorodi, 2024; Reid, 1996, p. 2; implied by Firesmith, 2015, p. 47; Souza et al., 2017, p. 3; Barbosa et al., 2006, p. 3)","Testing based on ""a model of the states the test item can occupy, the transitions between states, the events which cause transitions and the actions that can result from the transitions"" (IEEE, 2021c, p. 19)","Specification-based Testing (IEEE, 2022, pp. 9, 22; 2021c, pp. 5, 19, Fig. 2; Washizaki, 2024, p. 5-12; Hamburg and Mogyorodi, 2024; Sharma et al., 2021, Fig. 1; implied by Firesmith, 2015, p. 47; Peters and Pedrycz, 2000, Fig. 12.2), Model-based Testing (IEEE, 2022, pp. 9, 13; 2021a, pp. 6, 33; 2021b, pp. 3, 20; 2021c, pp. 6, 10, 19; Bourque and Fairley, 2014, p. 4-10; implied by Hamburg and Mogyorodi, 2024; Doğan et al., 2014, pp. 179, 184; Souza et al., 2017, p. 3), Web Application Testing (Doğan et al., 2014, p. 179), Closed-Loop Testing (Preuße et al., 2012, p. 6), State-based Testing, Open-Loop Testing?","STT (IEEE, 2021a, p. 33), Finite State Testing (Hamburg and Mogyorodi, 2024), Finite-State Machine (FSM) Testing (implied by Doğan et al., 2014, p. 179), Model Edge Testing?, Model Transition Testing? (implied by Doğan et al., 2014, p. 184)","Examples of state models include state tables (IEEE, 2022, p. 9; 2021c, pp. 5, 19), state [transition (2022, p. 9; 2021c, pp. 5, 19)] diagrams (2017, p. 438), and finite-state machines (Washizaki, 2024, p. 5-12); their states should be ""discrete, identifiable and finite in number"" (IEEE, 2021c, p. 19). See Chow (1978); Patton (2006, pp. 79-87)"
State-based Web Browser Testing,"Technique (Doğan et al., 2014, p. 193)",,"Web Application Testing (Kam, 2008, Tab. 1, p. 11), Formal Methods (Tab. 1, p. 12), Specification-based Testing, Control Flow Testing, Functional Testing (Tab. 1), State Transition Testing (p. 15), Transaction (Flow) Testing, Transaction (Flow) Verification (p. 16)","State-based Testing (Doğan et al., 2014, p. 193), Web Browser Testing (implied by Kam, 2008, pp. 14-15), Statechart Testing? (implied by Kam, 2008, Tab. 1, p. 11)","""Needs to be integrated with another testing method""  (Kam, 2008, p. 15) and ""can test whether or not there are any side effects caused by users using the back, forward, and refresh/reload control buttons to navigate the website"" (pp. 15-16) although ""the reload path branches [are often pruned] to eliminate redundancy"", which overlooks ""the security of time sensitive [sic] expiration pages"" (p. 16). See Di Lucca et al., [22, 23]"
Statement Testing,"Technique (IEEE, 2022, pp. 9, 22; 2021c, pp. 5, 23, Fig. 2; 2017, p. 438; Washizaki, 2025, p. 5-13; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 49; Reid, 1996, p. 2)","Testing ""based on exercising executable statements"" (IEEE, 2022, p. 9; 2021c, p. 5; similar on p. 23; Lyu, 1996, p. 533; Patton, 2006, p. 119; Kam, 2008, p. 48) by ""forc[ing the] execution of individual statements in a test item"" (IEEE, 2017, p. 439; OG IEEE, 2013)","Structure-based Testing (IEEE, 2022, p. 22; 2021c, pp. 5, 23, Fig. 2; Hamburg and Mogyorodi, 2024; Lyu, 1996, p. 533; Sharma et al., 2021, Fig. 1), Control Flow Testing (IEEE, 2021c, p. 23; Washizaki, 2025, p. 5-13; Firesmith, 2015, p. 49; Patton, 2006, pp. 118-119; van Vliet, 2000, p. 421; Reid, 1996, p. 3), Model-based Testing (IEEE, 2021c, p. 23), Automated Testing (can be supported with ""statement coverage analysers"") (p. 10; similar in Patton, 2006, pp. 119-120), Path Testing (implied by Lyu, 1996, p. 533), Branch Testing (Patton, 2006, p. 121; implied by Peters and Pedrycz, 2000, p. 481; OG Miller et al., 1994; van Vliet, 2000, Fig. 13.17; Reid, 1996, Fig. 2), Branch Condition Combination Testing (Patton, 2006, p. 121), Structure-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3), Unit Testing (if ""less than 5000 lines of code"") (p. 481; OG Miller et al., 1994), Developer Testing (implied by IEEE, 2021c, p. 23), Decision Testing (implied by Fig. F.1), Structural Testing (implied by Reid, 1996, p. 6)","Line Testing (Patton, 2006, pp. 118-119; implied by Doğan et al., 2014, Tab. 11), All-Nodes Testing (van Vliet, 2000, p. 421), Node Testing (implied by Doğan et al., 2014, Tab. 11)","Requires 100% coverage to be effective (Peters and Pedrycz, 2000, p. 481; OG Miller et al., 1994) but doesn't guarantee correctness (Patton, 2006, p. 119; van Vliet, 2000, p. 421). See Myers (1979)"
Static Analysis,Approach,"The ""process of evaluating a system or component based on its form, structure, content, or documentation"" (IEEE, 2017, p. 439) that ""involves the use of tools to detect anomalies in code or documents without execution"" (IEEE, 2022, p. 18)","Static Testing (IEEE, 2022, pp. 9, 17, 25, 28; 2021a, pp. 3, 21; Hamburg and Mogyorodi, 2024; Gerrard, 2000a, Fig. 4, p. 12; 2000b, p. 3), W-Model Testing (Gerrard, 2000a, Fig. 4), Developer Testing (implied by Lahiri et al., 2013, p. 345)",,See ISO/IEC 20246
"Static Assertion Checking (Lahiri et al., 2013, p. 345)",Practice?,"The use of ""logical techniques … to prove, before runtime, that no violations of [separation logic] specifications will take place at runtime"" (Chalin et al., 2006, p. 343)","Assertion Checking, Static Analysis (Lahiri et al., 2013, p. 345; Chalin et al., 2006, p. 343), Specification-based Testing?","Static Verification (incorrectly) (implied by Chalin et al., 2006, p. 343), SAC?","Often limited by ""the need to define a[ ""fairly complete"" (Chalin et al., 2006, p. 343)] assertion (or specification) to check, to provide environment specifications and to provide auxiliary invariants for loops and procedures"" (Lahiri et al., 2013, p. 345), but ""often provides stronger guarantees and … can give them earlier"" than RAC (Chalin et al., 2006, p. 343)"
Static Testing,"Level (IEEE, 2021b, p. 43)","Testing ""in which a test item … is examined against a set of quality or other criteria without the test item being executed"" (IEEE, 2022, p. 9; 2021a, p. 3; 2017, p. 440; OG 2013)","W-Model Testing (Gerrard, 2000a, p. 9), Specification-based Testing? (implied by Chalin et al., 2006, p. 343)","Static Analysis (Peters and Pedrycz, 2000, p. 438), Static Verification? (implied by Chalin et al., 2006, p. 343)","""Helps form an optimal test strategy"" (IEEE, 2022, p. 21), ""can be performed anywhere in the life cycle"" (IEEE, 2022, p. 17), including ""prior to dynamic testing[,] and can find defects before test execution becomes possible"" (IEEE, 2022, p. 21). Not always considered ""testing"" (Washizaki, 2024, p. 5-2). See ISO/IEC 20246"
Statistical Testing,"Technique (Kam, 2008, pp. 23, 48)","Testing ""in which a model of the statistical distribution of the input is used to construct representative test cases"" (Kam, 2008, p. 48)","Usage-based Testing (Washizaki, 2024, p. 5-15), Model-based Testing",,"""Often provide[s] a snapshot of the more troublesome areas of the software product under examination"" (Washizaki, 2024, p. 12-8). ""Usage-based statistical testing is applied more during the acceptance testing stage"" (Washizaki, 2024, p. 5-15); how to track this? Implied to be related to random testing (Washizaki, 2024, p. 5-15)"
"Stepwise Abstraction (van Vliet, 2000, pp. 419-420)",Technique,,"Structural Analysis? (van Vliet, 2000, pp. 419-420)",,
Stress Testing,"Type (IEEE, 2022, pp. 9, 22; 2017, p. 442; implied by Firesmith, 2015, p. 54), Technique (IEEE, 2021c, p. 38-39)","Testing of ""the SUT [at or (Hamburg and Mogyorodi, 2024)] beyond its capabilities"" (Washizaki, 2024, p. 5-9; similar in IEEE, 2021c, p. 39) or ""under less-than-ideal conditions"" (Patton, 2006, p. 86) by ""providing extreme test conditions"" (Moghadam, 2019, p. 1187; similar in IEEE, 2021c, p. 39) ""to find the performance breaking points"" by ""manipulating … factors affecting the performance"" (Moghadam, 2019, p. 1187); for example, ""load[s] greater than what the system is expected to handle"" (Washizaki, 2024, p. 5-9; similar in IEEE, 2021c, p. 39) or insufficient available resources (IEEE, 2022, p. 9; 2021c, p. 39; 2017, p. 442; OG 2013), such as memory (IEEE, 2021c, p. 39; Hamburg and Mogyorodi, 2024), processors, disks (IEEE, 2021c, p. 39), or servers (Hamburg and Mogyorodi, 2024)","Performance-related Testing (IEEE, 2022, p. 22; 2021c, p. 38), Performance Efficiency Testing (IEEE, 2022, p. 9; 2017, p. 442), Model-based Testing, Requirements-based Testing, Conformance Testing (p. 38), Non-functional Testing (Washizaki, 2025, p. 5-9), Performance Testing (Hamburg and Mogyorodi, 2024; Koomen et al., 2006, p. 50; Moghadam, 2019, p. 1187; Gerrard, 2000b, p. 23), Capacity Testing (Firesmith, 2015, p. 54), Boundary Condition Testing (Patton, 2006, p. 86; implied by IEEE, 2017, p. 442), Negative Testing (Patton, 2006, p. 86), Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5)",,"Can be generated ""based on source code or system model analysis, or use-case based design approaches""; issues emerge from ""performance bottlenecks"" resulting from ""application-, platform-, … [or]  workload-wise causes"" (Moghadam, 2019, p. 1187)"
Strong Mutation Testing,"Technique (van Vliet, 2000, pp. 428-429)","Mutation testing performed at the program level, so that a program and its mutant produce different results (van Vliet, 2000, p. 429)","Mutation Testing, Fault-based Testing (van Vliet, 2000, pp. 428-429), System Testing?",,
Structural Analysis,Technique,"""The process of carefully and methodically reviewing the software design, architecture, or code for bugs without executing it"" (Patton, 2006, p. 92)","Static Testing, Structure-based Testing (Patton, 2006, pp. 91-104), At-the-Beginning Testing (ideally) (p. 92)",,"Less common than specification-based testing (often avoided because of ""the misconception that it’s too time-consuming, too costly, or not productive"" (Patton, 2006, p. 92)), but often used for ""military, financial, factory automation, or medical software, … in a highly disciplined development model"" (p. 91) since it can ""find bugs that would be difficult to uncover or isolate with dynamic black-box testing"" and ""gives the team's black-box testers ideas for test cases to apply"" (p. 92). If it is done, it is largely ""done by the language compiler"" or separate tools (van Vliet, 2000, pp. 413-414)"
Structural Testing,"Technique (Barbosa et al., 2006, p. 3)","Testing based on knowledge of ""what the code does and how it works"" gained by seeing it run (Patton, 2006, p. 106)","Dynamic Testing (Patton, 2006, pp. 105-121), Structure-based Testing (pp. 105-121; Peters and Pedrycz, 2000, p. 447)","Structure Testing (implied by Peters and Pedrycz, 2000, p. 447)",
Structure-based Testing,"Technique (IEEE, 2022, p. 22; 2021b, p. 45; 2021c, p. 8; Washizaki, 2025, pp. 5-10, 5-12; Hamburg and Mogyorodi, 2024; Firesmith, 2015, pp. 46, 49; Souza et al., 2017, p. 3; Sakamoto et al., 2013, p. 344; implied by IEEE, 2022, pp. 2, 4, 6, 9; Barbosa et al., 2006, p. 3), Type (Hamburg and Mogyorodi, 2024)","Testing ""derived from an examination of the structure[, internal contents or implementation (IEEE, 2017, p. 199)] of the test item"" (IEEE, 2022, p. 9; similar in 2021c, p. 8)","Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, pp. 3, 20; 2021c, pp. 6, 10), Dynamic Testing (IEEE, 2022, pp. 9, 17; 2017, p. 444; Washizaki, 2025, p. 5-13; Sharma et al., 2021, Fig. 1), Control Flow Testing (usually) (IEEE, 2021c, p. 10), Automated Testing (can be supported with ""statement coverage analysers""; IEEE, 2021c, p. 10), Static Testing (likely when supported by tools in IEEE, 2021c, p. 10; Washizaki, 2025, p. 5-13)","White-Box Testing (IEEE, 2022, p. 9; 2021c, p. 8; 2017, pp. 443-444; Washizaki, 2025, p. 5-10; Hamburg and Mogyorodi, 2024; Firesmith, 2015, pp. 46, 49 (without hyphen); Sakamoto et al., 2013, p. 344; Patton, 2006, p. 55), Glass-Box Testing (IEEE, 2022, p. 9; 2017, pp. 443-444; Washizaki, 2025, p. 5-10; Hamburg and Mogyorodi, 2024; Peters and Pedrycz, 2000, p. 439), Structural Testing (IEEE, 2022, p. 9; 2017, pp. 443-444; Hamburg and Mogyorodi, 2024; implied by Barbosa et al., 2006, p. 3), Code-based Testing (Washizaki, 2025, p. 5-12; Bourque and Fairley, 2014, p. 4-8; Kam, 2008, p. 43), Clear-Box Testing (IEEE, 2021c, p. 8; Washizaki, 2025, p. 5-10; Hamburg and Mogyorodi, 2024; Patton, 2006, p. 55), Coverage Testing (Lyu, 1996, p. 532)","""Can be performed at different levels (such as code development, code inspection, or unit testing)"" (Washizaki, 2025, p. 5-13) but ""can only detect errors of commission"" (IEEE, 2021c, p. 138). ""Less dependent on the tester than black-box testing because it is based on the implementation"" (Sakamoto et al., 2013, p. 346). Hamburg and Mogyorodi (2024), Umar (2020), and Kam (2008) have more synonyms; it seems like ""structure-based testing"" is specific to dynamic testing (IEEE, 2017, p. 444) but ""structural testing"" isn't (p. 443)"
"Structure-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",Approach,,"Implementation-oriented Testing (Peters and Pedrycz, 2000, Fig. 12.3)",,
Structured Scripting,"Technique (implied by Hamburg and Mogyorodi, 2024)","""Scripting … that builds and utilizes a library of reusable (parts of) scripts"" (Hamburg and Mogyorodi, 2024)","Scripted Testing, Structured Testing",,
"Structured Testing (Hamburg and Mogyorodi, 2024)",Practice?,,,,
Structured Walkthroughs,Practice?,"""Systematic examination[s] of the requirements, design, or implementation of a system, or any part of it, by qualified personnel"" (IEEE, 2017, p. 444; OG ISO/IEC, 2015)","Walkthroughs, Structured Testing, Static Testing",,
Stuck Key Testing,"Technique (Firesmith, 2015, p. 51)",,"Random Testing (Firesmith, 2015, p. 51)",,
Sub-boundary Condition Testing,Technique,"Testing boundary conditions ""internal to the software [but not] … necessarily apparent to an end user"" (Patton, 2006, p. 75), such as powers of two (pp. 75-76), ASCII and Unicode tables (pp. 76-77), or ""complex"" requirements, where behaviour depends on multiple conditions (van Vliet, 2000, p. 430)","Data Testing (Patton, 2006, pp. 75-77, 115), Structure-based Testing (p. 115), Boundary Condition Testing? (pp. 75-77)",,"Likely of interest for scientific computing and depends on knowledge of various data types (Smith and Carette, 2023)"
"Subcontractor Testing (Firesmith, 2015, p. 37)",Practice?,,"Development Organization Testing (Firesmith, 2015, p. 37)",,
"Subsystem Testing (Firesmith, 2015, p. 23)",Level (inferred from system testing),,"System Testing (Firesmith, 2015, p. 23), Unit Testing? (seems incorrect) (implied by Firesmith, 2015, p. 28)","Subsystem-based Testing? (Firesmith, 2015, p. 28)",
Summative Evaluation(s),Approach,"""Evaluation[s] designed and used to gather conclusions about the quality of a component or system, especially when a substantial part of it has completed design"" (Hamburg and Mogyorodi, 2024)",System Testing?,,
Survivability Testing,"Technique (Ghosh and Voas, 1999, p. 39), Type (implied by its quality (ISO/IEC, 2011); inferred from robustness testing and security testing)","Testing the ""degree to which a product or system continues to fulfill its mission by providing essential services in a timely manner in spite of the presence of [""anomalous events and malicious"" (Ghosh and Voas, 1999, p. 39)] attacks"" (ISO/IEC, 2011), often focusing on software infrastructure (Ghosh and Voas, 1999, p. 39) and/or networks (p. 41)","Recoverability Testing (ISO/IEC, 2011), Security Testing (2011, implied by Ghosh and Voas, 1999, p. 40), Specification-based Testing (can be, when source code is unavailable) (p. 39)",,"May be supported by fault injection (Ghosh and Voas, 1999, p. 39)"
Symbolic Execution,"Technique (IEEE, 2017, p. 451)","Testing ""in which program execution is simulated using symbols … and program outputs are expressed as logical or mathematical expressions involving these symbols"" (IEEE, 2017, p. 451)","Structure-based Testing, Static Testing (Peters and Pedrycz, 2000, Tab. 12.1), Static Analysis?, Correctness Testing?",,See Păsăreanu and Visser (2009)
"Synchronous Testing (Jard et al., 1999)",Practice?,"Testing that occurs in real time, so that, for example, ""an IUT can refuse an event and … the tester can observe the refusal"" (Jard et al., 1999, p. 26; OG [1])",,,"Sometimes possible to make asynchronous (e.g., by using ""logical stamps"" (Jard et al., 1999, p. 25), which are ""simple counting mechanism[s]"" (p. 27)) but this transformation is ""a difficult task"" (p. 26)"
"Syntactic Testing (implied by Peters and Pedrycz, 2000, Tab. 12.2)",Approach,"Testing where the ""program [is] treated as a hierarchy of syntactic elements determined by the grammar of the programming language"" (Peters and Pedrycz, 2000, Tab. 12.2)","Static Analysis, Static Testing",,"May support statement testing and branch testing (Peters and Pedrycz, 2000, Tab. 12.2)"
Syntax Testing,"Technique (IEEE, 2022, p. 22; 2021c, pp. 5, 14, Fig. 2; Washizaki, 2024, p. 5-11; Firesmith, 2015, p. 47; Kam, 2008, p. 48; Reid, 1996, p. 2)","Testing based on ""rules, where each rule defines the format of an input parameter in terms of 'sequences of', 'iterations of', or 'selections between' elements in the syntax"" (IEEE, 2021c, p. 14; similar in Peters and Pedrycz, 2000, p. 448; Reid, 1996, p. 2), which ""may be represented in a textual or diagrammatic format"" (IEEE, 2021c, p. 14)","Specification-based Testing (IEEE, 2022, p. 22; 2021c, pp. 5, 14, Fig. 2; Washizaki, 2024, p. 5-11; Firesmith, 2015, p. 47; Kam, 2008, p. 48; implied by Intana et al., 2020, p. 260; Peters and Pedrycz, 2000, Fig. 12.2), Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, p. 3; 2021c, pp. 6, 14; implied by Bourque and Fairley, 2014, p. 4-10), Formal Testing (IEEE, 2021c, p. 14; Washizaki, 2024, p. 5-11), Negative Testing (IEEE, 2021c, p. 14; most effectively in Reid, 1996, p. 2), Experience-based Testing (can be for the negative version) (IEEE, 2021c, p. 14; implied by Reid, 1996, p. 3), Positive Testing (IEEE, 2021c, p. 14), One-to-One Testing (can be), Minimized Testing (can be) (p. 15), Data Testing (Peters and Pedrycz, 2000, pp. 448-449), Grey-Box Testing (can be) (Patton, 2006, p. 220), Static Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1; implied by Washizaki, 2024, p. 1-10), Smoke Testing, Desktop Development Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Functional Testing? (Washizaki, 2024, p. 5-11), Web Application Testing (implied by Kam, 2008, p. 9; Gerrard, 2000b, pp. 3-5)","Formal Specification-based Testing (Washizaki, 2024, p. 5-11), Syntax-Driven Testing (Peters and Pedrycz, 2000, p. 448), Syntax Checking? (Fig. 12.2)","Positive inputs are called ""options"" and negative ones are ""mutations"" (IEEE, 2021c, p. 14; Reid, 1996, p. 2). ""There is currently no industry agreed approach to calculating coverage"", and doing so ""as a percentage is not possible … due to the potentially extremely large number of options and infinite number of mutations possible"" (p. 31; similar in Reid, 1996, p. 3). ""Permits automatic derivation of functional test cases"" (Washizaki, 2024, p. 5-11), such as from formal requirements (e.g., BDD scenarios and state models) (p. 1-20) and ""permit[s] desired properties of the specified software to be proved"" (p. 1-10). ""Provides an oracle for checking test results"" (p. 5-11), but ""determining an expected result is not always possible"" and may require ""additional business domain expertise"" (p. 1-20). ""Backus-Naur Form is commonly used for defining the syntax of test item inputs"" (IEEE, 2021c, p. 5) ""in a textual format"", and ""abstract syntax tree[s] can be used to represent formal syntax diagrammatically"" (p. 14). See Beizer 1995; Burnstein 2003; Intana et al., 2020, p. 260"
"Sys Admin Testing (Firesmith, 2015, p. 39)",Practice?,,"Operator Testing (Firesmith, 2015, p. 39)",,
System Integration Testing,"Level (IEEE, 2022, pp. 12, 22; 2021a, p. 6; 2021b, pp. 41, 43, 51; 2021c, p. 6; Hamburg and Mogyorodi, 2024; Black, 2009, pp. 14, 18, 62, 178)","Testing the ""progressive assembling of system components into the whole system"" (IEEE, 2017, p. 454; OG ISO/IEC, 2015), including between packages and ""external organizations (e.g. [sic] Electronic Data Interchange, Internet)"" (Kam, 2008, p. 48)","Integration Testing (Hamburg and Mogyorodi, 2024), System Testing (Firesmith, 2015, p. 23), Independent Tester Testing (can be) (IEEE, 2021b, p. 52), SoS Testing (implied by Black, 2009, Fig. 1-2)","SIT (implied by IEEE, 2021b, p. 54)","May be done in a system test environment (IEEE, 2021b, p. 42)"
System Qualification Testing,Level?,"""Testing performed on the completed, integrated system of software components, hardware components, and mechanics to provide evidence for compliance with system requirements and that the complete system is ready for delivery"" (Hamburg and Mogyorodi, 2024)","System Testing?, Dynamic Testing?",System Testing?,Difference between this and system testing?
System Testing,"Level (IEEE, 2022, pp. 12, 20-22, 26-27; 2021a, pp. 6, 11; 2021b, pp. 41, 43, 51; 2021c, p. 6; 2017, p. 467; 2016, p. 4; Washizaki, 2024, p. 5-7; Hamburg and Mogyorodi, 2024; Black, 2009, pp. 14, 24, 62, 178; Peters and Pedrycz, 2000, Tab. 12.3; van Vliet, 2000, p. 439; Hetzel, 1988, pp. 9-11, 134; Souza et al., 2017, p. 3; Sakamoto et al., 2013, p. 343; Barbosa et al., 2006, p. 3; Gerrard, 2000a, p. 13)","""Testing conducted on a complete, integrated system to evaluate [its] compliance with its specified requirements"" (IEEE, 2017, p. 456; OG 2012; similar in Hamburg and Mogyorodi, 2024; Peters and Pedrycz, 2000, Tab. 12.3; van Vliet, 2000, p. 439; Sakamoto et al., 2013, pp. 343-344), usually in regards to ""non-functional system requirements"", and verify its behaviour (Washizaki, 2024, p. 5-7; similar in Peters and Pedrycz, 2000, Tab. 12.3)","Dynamic Testing (IEEE, 2021a, pp. 11, 29-30; Gerrard, 2000a, p. 9), SoS Testing (IEEE, 2019b), Requirements-based Testing (IEEE, 2017, p. 456; OG 2012; Washizaki, 2024, p. 5-7; Peters and Pedrycz, 2000, Tab. 12.3, Fig. 12.41; van Vliet, 2000, p. 439), System Testing (Firesmith, 2015, p. 23), Specification-based Testing (van Vliet, 2000, p. 439; Hetzel, 1988, p. 15), V-Model Testing (Gerrard, 2000a, p. 9), W-Model Testing (Figs. 3-5), Independent Tester Testing (can be) (IEEE, 2021b, p. 52), Random Testing (often), Reliability Testing (can be) (van Vliet, 2000, p. 439), Integration Testing (implied by Sakamoto et al., 2013, p. 344)","Story Testing (IEEE, 2021b, p. 41), Product Testing (implied by Lyu, 1996, p. 532)","May be done in a system test environment (IEEE, 2021b, p. 42) and can be supported for distributed systems by ""conventional capture/replay and database editing tools"" (Sneed and Göschl, 2000, p. 3; OG Beizer, 1995)"
Systems Integration Testing,Level?,"""Testing conducted on multiple complete, integrated systems to evaluate their ability to communicate successfully with each other and to meet the overall integrated systems' specified requirements"" (IEEE, 2017, p. 457)",Integration Testing,"Large Scale Integration Testing (Gerrard, 2000a, p. 13)",
Systems Testing,"Level (Black, 2009, p. 14; Hetzel, 1988, pp. 122, 134-139, 149)",,"SoS Testing (implied by Black, 2009, Fig. 1-2)","System Testing (implied by Hetzel, 1988, pp. 73, 121-122, 132-134, 139, 280)",
"Tailored Conformance Testing (inferred from tailored conformance (IEEE, 2021a, pp. 9-10; 2021b, pp. 5, 17, 37; 2021c, p. 7) and conformance testing)",Type (inferred from conformance testing),"Testing that ""the chosen subset of requirements from the chosen (non-empty) set of techniques and corresponding test coverage measurement approaches have been satisfied"", including justification and ""rationale, including the consideration of any applicable risks"" as to why ""the normative requirements of a technique … or measure … are not followed completely"" (IEEE, 2021c, p. 7; may be specific to this)",Conformance Testing,,"Should ""be agreed [upon] by the relevant stakeholders"" (IEEE, 2021c, p. 7)"
"Technical Reviews (Washizaki, 2024, p. 12-14)",Approach,"The ""systematic evaluation of a software product by a team of qualified personnel that examines the suitability of the software product for its intended use and identifies discrepancies from specifications and standards"" (IEEE, 2017, p. 463; OG 2008), often done ""at logical transition points in a system life cycle"" (2017, p. 463; OG 2014)","Formal Reviews (Hamburg and Mogyorodi, 2024)",,"""Can … be more focused and address a specific project phase"" (Washizaki, 2024, p. 12-14; OG [24]). More types given by Washizaki (2024, p. 12-14)"
"Technical Testing (inferred from technical requirements (IEEE, 2017, p. 463) and requirements-based testing)",Approach,"Testing ""requirements relating to the technology and environment, for the development, maintenance, support and execution of the software"" (IEEE, 2017, p. 463)","Requirements-based Testing, Non-functional Testing?",,"Includes the ""programming language, testing tools, operating systems, database technology and user interface technologies"" (IEEE, 2017, p. 463)"
"Template Variable Testing (Doğan et al., 2014, Tab. 13; Sakamoto et al., 2013)",Approach,"Testing that ""focuses on the … template variables … important for testing a web application's functionality related to the dynamic content of an HTML document"" (Sakamoto et al., 2013, p. 344)","Web Application Testing (Doğan et al., 2014, Tab. 13; Sakamoto et al., 2013, pp. 343-344), Integration Testing (pp. 343, 347), Grey-Box Testing (p. 347)",,"""Allows … [for] exhaustive gray-box integration testing"" (Sakamoto et al., 2013, p. 347) but ""cannot assess input values themselves"" (p. 356)"
"Test Browsing (Gerrard, 2000a, p. 12, Tab. 2; 2000b, Tab. 1)",Approach,"Testing that ""aims to address … faults that relate to the navigation through web pages, the availability of linked objects"", download speeds, and ""integration of web pages to server-based components"" (Gerrard, 2000b, p. 9)","Integration Testing (partially) (Gerrard, 2000b, p. 9), Web Application Testing",,
"Test Environment Testing (Firesmith, 2015, p. 25)",Technique?,,,,
"Test Tool Testing (Firesmith, 2015, p. 25)",Technique?,,,,
Test-driven Development,Process,"A development process where ""developers write the tests before they develop their code"" (Sangwan and LaPlante, 2006, p. 25), ""think[ing] of ways to break [their code] through testing, and then add[ing] code to address that situation"" (p. 26)","Developer Testing (Sangwan and LaPlante, 2006, p. 25), Robustness Testing, Fault Tolerance Testing (p. 26), Incremental Testing (p. 29)",,"Tends to focus on ""low-level unit testing"" (Sangwan and LaPlante, 2006, p. 25), contributing to ""difficulty with integration and system-level testing, poor confidence in code quality despite high code coverage provided by unit tests that pass trivially, and increased effort in maintaining test code as ratio of test to production code goes up"" (p. 28). However, it ""can improve code quality"" (p. 25), especially when ""the quality of unit tests"" is improved and continuous/automated testing (p. 28) and test frameworks are used, and tends to improve communication in an organization (p. 29)"
"Tester Testing (Firesmith, 2015, p. 39)",Practice?,,"Role-based Testing (Firesmith, 2015, p. 39)",,
TestUML Testing,Approach,"Testing using ""class and state diagrams via static and dynamic analysis of [the] web application[]"" (so it doesn't require documentation (Kam, 2008, p. 12)) generated by WebUml ""to define test cases and choose[] test paths based on random walk analysis"" (p. 11; OG [4, 5])","Web Application Testing, Formal Methods, Structure-based Testing, Control Flow Testing, Functional Testing (Kam, 2008, Tab. 1), Random Walk Testing? (Kam, 2008, p. 12), UML Testing",,"These ""class and state diagrams"" are generated using ""UML and statecharts"" and then ""used as an oracle""; ""can potentially be a complete testing method to perform fully automatic V&V processes"", but cannot reveal ""dead code or … Trojan horse program[s]"" or ""test problems such as uncontrolled flow transaction[s] and syntax error[s]"" (Kam, 2008, p. 12)"
TestWeb Testing,Approach,,"Web Application Testing, UML Testing, Structure-based Testing, Control Flow Testing, Functional Testing (Kam, 2008, Tab. 1)",,
"Textual Testing (implied by Peters and Pedrycz, 2000, Tab. 12.2)",Approach,"Testing where the ""program code [is] treated as a sequence of characters or tokens"" (Peters and Pedrycz, 2000, Tab. 12.2)","Static Analysis, Static Testing",,"""Text editors, line counters, [and] scanners support this view"" (Peters and Pedrycz, 2000, Tab. 12.2), as do linters, presumably"
Think Aloud Usability Testing,"Technique (Hamburg and Mogyorodi, 2024)","Testing ""where test participants share their thoughts with the moderator and observers by thinking aloud while they solve usability test tasks"" which helps the testers ""understand the test participant"" (Hamburg and Mogyorodi, 2024)",Usability Testing,,
Three-Value Boundary Testing,"Technique (inferred from boundary value analysis and inclusion in IEEE, 2021c, p. 13)","Testing ""values on the boundary and an incremental [""the smallest significant value for the data type""] distance each side of the boundary of the equivalence partition"" for ""each boundary"" (IEEE, 2021c, p. 13)","Boundary Value Analysis (IEEE, 2021c, p. 13)",,"""Can be required for certain circumstances"", such as those requiring ""rigorous testing"" (IEEE, 2021c, p. 13)"
Threshold Testing,Approach,"Testing based on ""the coverage of neurons exceeding a threshold activation value in a neural network for a set of tests"" (Hamburg and Mogyorodi, 2024)",ML Model Testing,,
"Time-domain-based Testing (implied by Lyu, 1996, pp. 532, 539)",Approach,,"Model-based Testing (Lyu, 1996, p. 539)",,See Musa et al. (1987)
Top-Down (Integration) Testing,Level (inferred from integration testing),"Integration testing that ""starts with the highest-level component[s] … and proceeds through progressively lower-levels"" (IEEE, 2017, p. 479; similar in van Vliet, 2000, p. 438) ""with lower level components being simulated by stubs"" (Kam, 2008, p. 48; Patton, 2006, pp. 109-110; Peters and Pedrycz, 2000, Tab. 12.8; van Vliet, 2000, p. 438)","Integration Testing (Washizaki, 2024, p. 5-7; Sharma et al., 2021, p. 603; Kam, 2008, p. 48; Patton, 2006, pp. 109-110; Sangwan and LaPlante, 2006, p. 27; Peters and Pedrycz, 2000, p. 488, Tab. 12.8; van Vliet, 2000, p. 438), Sandwich Integration Testing (Peters and Pedrycz, 2000, Tab. 12.8), Incremental Testing (Kam, 2008, p. 48)",,"Difference between this as integration testing vs. not (as in Firesmith, 2015, p. 28)?"
Tours,"Practice (IEEE, 2022, p. 34)","Testing that is quite general and ""guides testers through the paths of an application like a tour guide leads a tourist through the landmarks of a big city"" (IEEE, 2022, p. 34), or that is exploratory yet ""organized around a special focus"" (Hamburg and Mogyorodi, 2024)","Exploratory Testing (IEEE, 2022, p. 34; Hamburg and Mogyorodi, 2024), Experience-based Testing (IEEE, 2022, pp. 4, 34; 2021c, p. 4)","Landmark Tours (implied by IEEE, 2022, p. 34)",
Transaction Flow Testing,"Technique (inferred from scenario testing and IEEE, 2021c, p. 20)",,"Scenario Testing (IEEE, 2021c, p. 20)","Transaction Testing? (Gerrard, 2000a, Fig. 5)",See Beizer 1995
Transaction Testing,Approach,"Testing that ""addresses the integration of the browser pages, web server and other server-based components"" (Gerrard, 2000b, p. 13) ""to ensure that the entire transaction is processed correctly"", including ""the direct and indirect interfaces"", and transfer of control and data (p. 16)","Dynamic Testing (Gerrard, 2000a, Fig. 5?, Tab. 2), Functionality Testing, System Testing, Functional Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Web Application Testing? (Kam, 2008, p. 9), W-Model Testing? (Gerrard, 2000a, Fig. 5), Integration Testing (2000b, pp. 13, 16), End-to-end Functionality Testing (p. 16)",,"Should be automated if there are complex components or custom-built software involved (Gerrard, 2000b, p. 17)"
Transaction Verification,Approach,"Testing that ""aims at ensuring that … the correct forms handler is invoked and that the parameters passed to the forms handler are correct"" (Gerrard, 2000b, p. 11)","Smoke Testing, Test Browsing, Static Testing (Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), Web Application Testing? (Kam, 2008, p. 9), Integration Testing (implied by Gerrard, 2000b, p. 16)","Transaction Flow Verification? (Gerrard, 2000a, Fig. 5)","""Not the same as Application System Testing""; ""focuses … on selected test cases that exercise particular interfaces between components in the technical architecture and scenarios (in volumes) that System tests might not address"" (Gerrard, 2000b, p. 16)"
"Translation Validation (Lahiri et al., 2013, p. 354; OG [26], [23], [18])",Technique?,,"Equivalence Checking? (Lahiri et al., 2013, p. 354; OG [26], [23], [18])",,
t-wise Testing,"Technique (Washizaki, 2025, p. 5-11)","Testing that ""considers every possible combination of [some] t input …[,] including higher-level combinations"" (Washizaki, 2025, p. 5-11)","All Combinations Testing (Washizaki, 2025, p. 5-11)","Orthogonal Array Testing (Washizaki, 2025, p. 5-11; implied by Valcheva, 2013, p. 473)",
Two-Value Boundary Testing,"Technique (inferred from boundary value analysis and inclusion in IEEE, 2021c, p. 13)","Testing ""values on the boundary and an incremental [""the smallest significant value for the data type""] distance outside the boundary of the equivalence partition"" for ""each boundary"" (IEEE, 2021c, p. 13)","Boundary Value Analysis (IEEE, 2021c, p. 13)",,"""Typically adequate in most situations"" (IEEE, 2021c, p. 13)"
"UI Testing (Doğan et al., 2014, Tab. 8)",Approach,,"Web Application Testing (Doğan et al., 2014, Tab. 8; implied by Kam, 2008, p. 9), Keyword-driven Testing (implied by IEEE, 2022, p. 5; 2016, p. 3)",,
"UML Model-based Testing (Doğan et al., 2014, p. 194)",Technique?,,Model-based Testing,"UML Testing? (implied by Kam, 2008, Tab. 1)",
Unit Testing,"Level (IEEE, 2022, pp. 12, 20-22, 26-27; 2021a, pp. 6, 11; 2021b, Fig. 2, pp. 41, 43, 51; 2021c, p. 6; 2017, p. 467; 2016, p. 4; Washizaki, 2025, p. 5-6; Hamburg and Mogyorodi, 2024; Lyu, 1996, p. 532; Black, 2009, pp. 12, 14, 18, 24; Peters and Pedrycz, 2000, Tab. 12.3; van Vliet, 2000, p. 438; Hetzel, 1988, pp. 9-10, 32, 73, 134; Souza et al., 2017, p. 3; Sakamoto et al., 2013, p. 343; Barbosa et al., 2006, p. 3), Technique (implied by Engström and Petersen, 2015, pp. 1-2)","""Testing of individual hardware or software components"" (IEEE, 2012, p. 8; similar in Hamburg and Mogyorodi, 2024; Lyu, 1996, p. 532; Sangwan and LaPlante, 2006, p. 26; van Vliet, 2000, p. 438) or of ""individual routines and modules"" (IEEE, 2017, p. 490) ""can include consideration of basic functionality"" (2022, p. 13; 2021a, p. 6; 2021c, p. 6)","Dynamic Testing (IEEE, 2021a, pp. 11, 29; Gerrard, 2000a, p. 9), Construction Testing (Washizaki, 2025, p. 4-7), V-Model Testing (Gerrard, 2000a, p. 9), W-Model Testing (Figs. 3-5), Development Testing (can be) (IEEE, 2021b, pp. 42, 52), Automated Testing (often) (Washizaki, 2024, p. 4-14), Functionality Testing, Smoke Testing (implied by IEEE, 2022, p. 13; 2021c, p. 6)","Component Testing (IEEE, 2022, pp. 12-13; 2021c, p. 6; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 22; Peters and Pedrycz, 2000, p. 444; Hetzel, 1988, p. 73), Module Testing (Hamburg and Mogyorodi, 2024; Patton, 2006, p. 109; Hetzel, 1988, p. 73), Program Testing (Kam, 2008, p. 46; implied by Hetzel, 1988, p. 73), Class Testing (Sneed and Göschl, 2000, p. 3)",
Unscripted Testing,Approach,"""Testing in which the tester's actions are not prescribed by written instructions in a test case"" (IEEE, 2022, p. 15; 2017, p. 490; OG IEEE, 2013)","Dynamic Testing (IEEE, 2022, p. 15; 2017, p. 490)",,
Usability Testing,"Type (IEEE, 2022, pp. 22, 26-27; 2021a, p. 8; 2021c, pp. 7, 40, Tab. A.1; Koomen et al., 2006, p. 9; implied by its quality (IEEE, 2015, p. 8; OG NATO AEP-67); Firesmith, 2015, p. 53)","Testing that ""evaluate[s] how easy it is for ['specified users' to use and (IEEE, 2021c, p. 40; 2017, p. 492; OG ISO/IEC, 2013)] learn to use the software"" [""to achieve assigned goals"" (IEEE, 2021c, p. 40)] by testing functionality (Washizaki, 2024, p. 5-10, IEEE, 2017, p. 493), documentation, and/or recovery (Washizaki, 2024, p. 5-10), as well as how ""attractive to the users"" the product is (Kam, 2008, p. 48)","Dynamic Testing (IEEE, 2021a, p. 29), Model-based Testing, Conformance Testing (2021c, p. 40), Requirements-based Testing (p. 40; Kam, 2008, p. 47), Security Testing (IEEE, 2015, p. 8; OG NATO AEP-67; partially in IEEE, 2017, pp. 404-405), Non-functional Testing (Bertolino et al., 2019, p. 93:11, Kam, 2008, p. 46; Gerrard, 2000a, Tab. 2; 2000b, Tab. 1), System Testing, Manual Testing, Usability Testing (2000a, Tab. 2; 2000b, Tab. 1), Automated Testing (can be) (IEEE, 2021b, p. 10), Regression Testing (can be) (Washizaki, 2024, p. 5-8)","Interaction Capability Testing (ISO/IEC, 2023a), Fitness-for-Use Testing (IEEE, 2017, p. 493), Human-Computer Interaction Testing? (Washizaki, 2024, p. 5-10)","""Usability requirements specify the usability goals …  based on test item goals … and the contexts of use for the test item"" (IEEE, 2021c, p. 40). Related to acceptance testing (van Vliet, 2000, p. 439) and potentially reusability (IEEE, 2017, p. 492), but this relation may just be syntactic. See ISO 9241-1; [20] in Gerrard, 2000b, p. 22"
"Usability Reviews (Hamburg and Mogyorodi, 2024)",Approach,,"Usability Testing, Reviews (Hamburg and Mogyorodi, 2024)",,
Usability Walkthroughs,Approach,"""Usability evaluation[s] in which one or more evaluators step through a scenario playing the role of a user and identifying usability problems associated with successful completion of the scenario[s]"" (IEEE, 2017, p. 493; OG ISO/IEC, 2010)","Usability Testing (IEEE, 2017, p. 493), Walkthroughs",,"""Evaluators can include usability specialists, developers, [and/or] end users"" (IEEE, 2017, p. 493)"
Usage-based Testing,"Technique (Washizaki, 2024, p. 5-15)","Testing that ""usually rel[ies] on a usage model or profiles"" in the context of ""the actual operational environment"" and ""usage by the target stakeholder"" (Washizaki, 2024, p. 5-15)","Operational Profile Testing (Hamburg and Mogyorodi, 2024), Online Testing?",,"May involve the development/implementation of an ""operational profile"" (Hamburg and Mogyorodi, 2024)"
Use Case Testing,"Technique (Kam, 2008, pp. 48-49)","""Testing sequences of interactions (i.e. scenarios) between the test item and actors"" where ""various actions are performed by the test item as a result of various triggers from the actors"" (IEEE, 2021c, p. 20)","Model-based Testing (IEEE, 2022, p. 13; 2021a, pp. 6, 33; 2021b, p. 3; 2021c, pp. 6, 20), Scenario Testing (p. 20; OG Hass, 2008), Specification-based Testing (Sharma et al., 2021, Fig. 1; Kam, 2008, pp. 48-49), User Story Testing?","Sometimes spelled with a hyphen (Sharma et al., 2021, Fig. 1), Scenario Testing (Hamburg and Mogyorodi, 2024), User Scenario Testing (2024; although ""an actor can be a user or another system"" (IEEE, 2021c, p. 20))","See Hass, 2008; Marchetto et al., 2008"
User Acceptance Testing,"Level (IEEE, 2022, p. 22; 2021b, pp. 43, 51; Firesmith, 2015, p. 30; Black, 2009, pp. 14, 24; inferred from acceptance testing)","""A type of acceptance testing performed to determine if intended users accept the system"" (Hamburg and Mogyorodi, 2024)","Acceptance Testing (IEEE, 2022, p. 22; Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 30), Customer Acceptance Testing (can be) (IEEE, 2021b, p. 46), SoS Testing (implied by Black, 2009, Fig. 1-2), User Testing","UAT (Hamburg and Mogyorodi, 2024; Firesmith, 2015, p. 30; implied by IEEE, 2021b, p. 54)","Also mentioned by Washizaki (2024, p. 6-13)"
"User as Tester Testing (Firesmith, 2015, p. 39)",Practice?,,"User Testing (Firesmith, 2015, p. 39), Dynamic Testing, Specification-based Testing, Behavioural Testing (Patton, 2006, pp. 87-88)",,"Behaving like (or getting feedback from) an inexperienced user can help ""discover bugs"" by not ""operating on any rules or making any assumptions"" (Patton, 2006, pp. 87-88)"
User Interface Navigation Testing,"Technique (Firesmith, 2015, p. 47)",,"Specification-based Testing (Firesmith, 2015, p. 47), UI Testing, User Testing, Procedure Testing?",,
"User Organization Testing (Firesmith, 2015, p. 37)",Practice?,,"Organization-based Testing (Firesmith, 2015, p. 37), User Testing",,
User Session Data Testing,Approach,,"Web Application Testing, User Session Testing, Specification-based Testing (usually), Control Flow Testing, Functional Testing (Kam, 2008, Tab. 1)",,
"User Session Testing (Kam, 2008, Tab. 1)",Approach,,User Testing,User-session-based Testing?,
User Story Testing,Approach,"Testing ""in which test conditions are the acceptance criteria of user stories"" or ""requirement[s] consisting of one sentence … capturing the functionality a user needs, the reason behind it, any non-functional criteria, and … acceptance criteria"" (Hamburg and Mogyorodi, 2024)","Specification-based Testing (Hamburg and Mogyorodi, 2024), System Testing, Requirements-based Testing, User Testing",,
User Surveys,Approach,"""Evaluation[s] whereby a representative sample of users are asked to report subjective evaluation into a questionnaire based on their experience in using a component or system"" (Hamburg and Mogyorodi, 2024)","Usability Testing, User Testing",,
"User Testing (Firesmith, 2015, p. 39)",Practice?,,"Role-based Testing (Firesmith, 2015, p. 39)",,
User-Agent Based Testing,Approach,"""Testing in which a test client is used to switch the user agent string and identify itself as a different client while executing test suites"" (Hamburg and Mogyorodi, 2024)",,,Could be used in security testing
User-based Evaluations,Approach,"""Evaluation[s] that involve[] representative users performing tasks with the system … focused on usability[,] … efficiency, effectiveness, [and/or] user satisfaction"" (IEEE, 2017, p. 498; OG ISO/IEC, 2016)","Usability Testing (IEEE, 2017, p. 498), User Testing, Acceptance Testing?",,
"User-initiated Built-In Testing (Firesmith, 2015, p. 31)",Practice?,,"Built-In Testing (Firesmith, 2015, p. 31)","UBIT (Firesmith, 2015, p. 31)",
User-oriented Testing,Approach,"Testing of ""the user view of the software"" (Lyu, 1996, p. 532)",User Testing,,
"User-session-based Testing (Doğan et al., 2014, p. 183)",Approach,,"Session-based Testing, User as Tester Testing?, User Story Testing?",,"Potentially complementary to structure-based testing (Doğan et al., 2014, p. 193; OG Elbaum et al., 2005)"
Validation Testing,Approach,"""Test[ing] to determine whether an implemented system fulfils its specified requirements"" (IEEE, 2017, p. 499; ISO/IEC, 2015)",,,"OG definition was about a specific test, not a test approach"
Verification Testing,Approach,"Testing to determine whether a system fulfils ""its specified requirements at a particular stage of its development"" (IEEE, 2017, p. 504; ISO/IEC, 2015)",,,"This seems to conflate the definition of ""verification"" with ""validation"""
Visual Browser Validation,Approach,"Testing visually ""that the appearance of [web]pages on … other browser[s] are acceptable"" (Gerrard, 2000b, p. 8)","Usability Testing, Manual Testing, Static Testing?, Dynamic Testing? (Gerrard, 2000a; Tab. 2; 2000b, Tab. 1), Cross-Browser Compatibility Testing (implied by 2000b, p. 8), Web Application Testing, Visual Testing",,
Visual Testing,Approach,"Testing of GUI objects (Hamburg and Mogyorodi, 2024; Bajammal and Mesbah, 2018, p. 193) ""rel[ying] exclusively on the application's appearance"" and ""a visual comparison between a number of initial screenshots"" (Bajammal and Mesbah, 2018, p. 193)","GUI Testing, Web Application Testing (Bajammal and Mesbah, 2018, p. 193)",,"May use ""image recognition"" (Hamburg and Mogyorodi, 2024) and is ""highly fragile compared to DOM-based methods"" (Bajammal and Mesbah, 2018, p. 193; OG [4, 5]) since ""it is difficult to assess what state is the canvas in at any given moment"" (p. 194). May use visual analysis (p. 194). See ISO 29119-11"
"V-Model Testing (Firesmith, 2015, p. 29)",Practice?,"Testing that occurs alongside the phases of the waterfall model in a ""layered and phased nature"" (Gerrard, 2000a, p. 9)","Waterfall Testing (Washizaki, 2024, p. 10-6; OG [3]; Firesmith, 2015, p. 29), Lifecycle-based Testing (Washizaki, 2024, p. 10-5; OG [2, 3, 10]), W-Model Testing? (implied by Gerrard, 2000a, p. 9)",,"See Washizaki (2024, p. 10-6; OG [3])"
Volume Testing,"Technique (IEEE, 2021c, p. 38-39), Type (implied by Firesmith, 2015, p. 54; inferred from performance-related testing)","Testing to assess ""the SUT's internal storage limitations and its ability to exchange data and information"" (Washizaki, 2024, p. 5-9) or ""process specified volumes of data (usually at or near maximum specified capacity)"" (IEEE, 2017, p. 508; OG 2013; similar in 2021c, p. 39; Kam, 2008, p. 49)","Performance-related Testing, Model-based Testing, Requirements-based Testing, Conformance Testing (IEEE, 2021c, p. 38), Performance Efficiency Testing (2017, p. 508; OG 2013), Non-functional Testing (Washizaki, 2025, p. 5-9), Capacity Testing (Firesmith, 2015, p. 54; implied by IEEE, 2017, p. 508; OG 2013), Dynamic Testing, W-Model Testing (Gerrard, 2000a, Fig. 5)",,
Vulnerability Scanning,"Technique (IEEE, 2021c, p. 40)","""The use of automated testing tools to scan a test item for signs of specific known vulnerabilities"" (IEEE, 2021c, p. 40)","Security Testing, Automated Testing (IEEE, 2021c, p. 40)",,
Walkthroughs,"Technique (IEEE, 2017, p. 508)","Evaluations ""in which a designer or programmer leads members of the development team and other interested parties through a segment of documentation or code, and the participants ask questions and make comments about possible errors, violation of development standards, and other problems"" (IEEE, 2017, p. 508; similar in Hamburg and Mogyorodi, 2024); uses test data to ""walk through"" the execution of the program (van Vliet, 2000, p. 416)","Static Analysis (IEEE, 2017, p. 508), Formal Reviews (Hamburg and Mogyorodi, 2024), Reviews (Hamburg and Mogyorodi, 2024; Patton, 2006, p. 95), Role-based Testing (can be) (Peters and Pedrycz, 2000, p. 484), Structure-based Testing, Static Testing (implied by Tab. 12.1)","Sometimes spelled with a hyphen (""walk-through""; IEEE, 2021b, p. 74; 2017, pp. 133, 439, 508), Structured Walkthroughs (Hamburg and Mogyorodi, 2024), Code Walkthroughs (implied by Peters and Pedrycz, 2000, Tab. 12.1)","A more structured walkthrough may have specific roles, such as presenter, coordinator, secretary, maintenance oracle, standards bearer, and user representative (Peters and Pedrycz, 2000, p. 484). See ISO 20246"
Waterfall Testing,Practice?,"Testing performed in the context of the waterfall model; i.e., after implementation but before installation and checkout, either strictly sequentially (Washizaki, 2024, p. 10-5) or ""possibly with overlap but with little or no iteration"" (IEEE, 2017, p. 509)","Lifecycle-based Testing (Washizaki, 2024, p. 10-5; OG [2, 3, 10]; Firesmith, 2015, p. 29)",,
Weak Mutation Testing,"Technique (van Vliet, 2000, pp. 428-429)","Mutation testing performed at the component level, so that a component and its mutant produce different results (van Vliet, 2000, p. 429)","Mutation Testing, Fault-based Testing (van Vliet, 2000, pp. 428-429), Strong Mutation Testing (implied by Fig. 13.17), Unit Testing",,"""Is often easier to establish"" than strong mutation testing (van Vliet, 2000, p. 429)"
Web Application Testing,Approach,"Testing of ""an application that is accessed via a web browser using the Hypertext Transfer Protocol (HTTP) over a network such as the Internet or intranet"" (Kam, 2008, p. 1)",Domain-specific Testing?,"WAT (Doğan et al., 2014, p. 175; Kam, 2008, p. 17; implied by Doğan et al., 2014, p. 176, 197), Web Testing (Kam, 2008, p. 16), Rich Internet Applications Testing? (implied by Doğan et al., 2014, p. 176; OG Amalfitano et al., 2010)","Difficult due to many factors (e.g., asynchronicity, heterogeneity, nondeterminism) (Doğan et al., 2014, pp. 174-175) and is often overlooked (Kam, 2008, p. 7; OG [12]). About 70% of websites in 2003 had defects (Doğan et al., 2014, p. 174; OG BIG-SF). References [14]-[16] in (Kam, 2008) may have more references for ""web testing"" synonym, along with sources used in other documents (Doğan et al., 2014; Choudhary et al., 2010)"
WebApp Slicing,Approach,,"Web Application Testing, Slicing, Structure-based Testing, Data Flow Testing, Functional Testing (Kam, 2008, Tab. 1)",,
W-Model Testing,Practice?,"The combination of ""static testing of early document or code deliverables and dynamic test stages of software deliverables"" (Gerrard, 2000a, p. 9)","Incremental Testing?, Continuous Testing?","Since only dynamic testing is considered for this research, the in-scope part of W-model testing is equivalent to V-model testing",