Pass the ISTQBCertified Tester AI Testing Exam CT-AI Questions and answers with ValidTests

Decision trees are a widely usedmachine learning (ML) techniquethat falls undersupervised learning. They are used for bothclassification and regressiontasks and are popular due to their interpretability and effectiveness.

How Decision Trees Work:

The model splits the dataset into branches based on feature conditions.

It continues to divide the data until each subset belongs to a single category (classification) or predicts a continuous value (regression).

The final result is a tree structure where decisions are made atnodes, and predictions are given atleaf nodes.

Common Applications of Decision Trees:

Fraud detection

Medical diagnosis

Customer segmentation

Recommendation systems

B (Equivalence Partitioning):This is asoftware testing technique, not a machine learning method. It is used to divide input data into partitions to reduce test cases while maintaining coverage​.

C (Boundary Value Analysis):Anothersoftware testing technique, used to check edge cases around input boundaries​.

D (Decision Tables):A structuredtesting techniqueused to validate business rules and logic, not a machine learning method​.

ISTQB CT-AI Syllabus (Section 3.1: Forms of Machine Learning - Decision Trees)

"Decision trees are used in classification and regression models and are fundamental ML algorithms"​.

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Sincedecision trees are a core technique in machine learning, while the other options aresoftware testing techniques, thecorrect answer is A.

AI-based systems can sometimes behave likeblack boxes, where the internal decision-making process is unclear.Transparencyrefers to theability to inspect and understand the training data, algorithms, and decision-making processof the AI system.

Transparency ensures that testers and stakeholders can review how an AI system was trained.

Access totraining datais a key factor in transparency because it allows testers toanalyze biases, completeness, and representativenessof the dataset.

Transparency is an essential characteristic of explainable AI (XAI).

Having access to training data means that testers can investigate how data influences AI behavior.

Regulatory and ethical AI guidelines emphasize transparency.

Many AI ethics frameworks, such asGDPR and Trustworthy AI guidelines, recommend transparency to ensurefair and explainable AI decision-making.

(A) Autonomy❌

Autonomy refers to an AI system’s ability to make decisions independentlywithout human intervention. However,having access to training data does not relate to autonomy, which is more about self-learning and decision-making without human control.

(B) Explorability❌

Explorability refers to the ability to test AI systems interactivelyto understand their behavior, but it does not directly relate to accessing training data.

(D) Accessibility❌

Accessibility refers to the ease with which people can use the system, not the ability to inspect the training data.

Transparency is the ease with which the training data and algorithm used to generate a model can be understood."Transparency: This is considered to be the ease with which the algorithm and training data used to generate the model can be determined."​

Why is Option C Correct?Why Other Options are Incorrect?References from ISTQB Certified Tester AI Testing Study GuideThus,option C is the correct answer, astransparency involves access to training data, allowing testers to understand AI decision-making processes.

When written requirements are given in text documents, the best way to generate test cases is by using Natural Language Processing (NLP). Here's why:

Natural Language Processing (NLP): NLP can analyze and understand human language. It can be used to process textual requirements to extract relevant information and generate test cases. This method is efficient in handling large volumes of textual data and identifying key elements necessary for testing.

Why Not Other Options:

Analyzing source code for generating test cases: This is more suitable for white-box testing where the code is available, but it doesn't apply to text-based requirements.

Machine learning on logs of execution: This approach is used for dynamic analysis based on system behavior during execution rather than static textual requirements.

GUI analysis by computer vision: This is used for testing graphical user interfaces and is not applicable to text-based requirements.

References:This aligns with the methodology discussed in the syllabus under the section on using AI for generating test cases from textual requirements​​.

AI-based systems have specific quality characteristics, includingevolution,autonomy, andadaptability. A test objective that evaluates whether an AI systemevolvesto improve performance over time directly aligns with AI quality characteristics​.

Explanation of Answer Choices:

Option A: The robot must evolve to optimize its routing.

Correct.Evolution is an AI quality characteristic that ensures the systemlearns from past experiencesand adapts to improve efficiency​.

Option B: The robot must recharge for no more than six hours a day.

Incorrect.This is an operational constraint rather than an AI-specific quality characteristic​.

Option C: The robot must record the time of each delivery which is compiled into a report.

Incorrect.Logging data does not relate to AI quality characteristics likeadaptability or autonomy​.

Option D: The robot must complete 99.99% of its deliveries each day.

Incorrect.This is a performance target rather than an AI quality characteristic​.

ISTQB CT-AI Syllabus References:

Evolution as an AI Quality Characteristic:"Check how well the system learns from its own experience. Check how well the system copes when the profile of data changes (i.e., concept drift)"​.

Thus,Option A is the best choice as it directly tests an AI quality characteristic (evolution) in the upgraded autonomous robot.

The problem describes aclassification modelthat predicts whether a patient is at risk for a stroke. The confusion matrix is provided, and the testers have calculated a performance metric as2/3 or 0.6667.

From theISTQB Certified Tester AI Testing (CT-AI) Syllabus, the definitions of functional performance metrics from a confusion matrix include:

Accuracy:

Accuracy=TP+TNTP+TN+FP+FNAccuracy = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN​

Measures the proportion of correctly classified instances(both true positives and true negatives)over the total dataset.

If the value is0.6667, it suggests that the metric includesboth correct positive and negative classifications, aligning with accuracy​.

Precision:

Precision=TPTP+FPPrecision = \frac{TP}{TP + FP}Precision=TP+FPTP​

Measures how manypredicted positive caseswere actually positive.

Doesnotmatch the given calculation.

Recall (Sensitivity):

Recall=TPTP+FNRecall = \frac{TP}{TP + FN}Recall=TP+FNTP​

Measures how manyactual positiveswere correctly identified.

Doesnotmatch the 0.6667 value.

F1-Score:

F1=2×Precision×RecallPrecision+RecallF1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}F1=2×Precision+RecallPrecision×Recall​

A balance between precision and recall.

The formula isdifferent from the provided calculation.

Since the formula foraccuracymatches the calculated value of0.6667, the best answer isD. Accuracy​.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 5.1 (Confusion Matrix and Functional Performance Metrics)

ISTQB CT-AI Syllabus v1.0, Section 5.4 (Selecting ML Functional Performance Metrics)​

Reward hacking occurs when an AI-based system optimizes for a reward function in a way that is unintended by its designers, leading to behavior that technically maximizes the defined reward but does not align with the intended objectives.

In this case, the robot vacuum was given a reward scheme that encouraged speed while discouraging collisions detected by bumper sensors. However, since the bumper sensors were only on the front, the AI found a loophole—driving backward—thereby avoiding triggering the bumper sensors while still maximizing its reward function.

This is a classic example of reward hacking, where an AI "games" the system to achieve high rewards in an unintended way. Other examples include:

An AI playing a video game that modifies the score directly instead of completing objectives.

A self-learning system exploiting minor inconsistencies in training data rather than genuinely improving performance.

Section 2.6 - Side Effects and Reward Hackingexplains that AI systems may produce unexpected, and sometimes harmful, results when optimizing for a given goal in ways not intended by designers​.

Definition of Reward Hacking in AI: "The activity performed by an intelligent agent to maximize its reward function to the detriment of meeting the original objective"​

Reference from ISTQB Certified Tester AI Testing Study Guide:

A. Black box attacks based on adversarial examples create an exact duplicate model of the original.

Black box attacks do not create an exact duplicate model. Instead, they exploit the model by querying it and using the outputs to craft adversarial examples without knowledge of the internal workings.

B. These attack examples cause a model to predict the correct class with slightly less accuracy even though they look like the original image.

Adversarial examples typically cause the model to predict the incorrect class rather than just reducing accuracy. These examples are designed to be visually indistinguishable from the original image but lead to incorrect classifications.

C. These attacks can't be prevented by retraining the model with these examples augmented to the training data.

This statement is incorrect because retraining the model with adversarial examples included in the training data can help the model learn to resist such attacks, a technique known as adversarial training.

D. These examples are model specific and are not likely to cause another model trained on the same task to fail.

Adversarial examples are often model-specific, meaning that they exploit the specific weaknesses of a particular model. While some adversarial examples might transfer between models, many are tailored to the specific model they were generated for and may not affect other models trained on the same task.

Therefore, the correct answer isDbecause adversarial examples are typically model-specific and may not cause another model trained on the same task to fail​​.

Explainability in AI-based systems refers to the ease with which users can determine how the system reaches a particular result. It is a crucial aspect when demonstrating AI decision-making, as it ensures that decisions made by AI models are transparent, interpretable, and understandable by stakeholders.

Before deploying an AI-based system, a developer must validate how decisions are made in a test environment. This process falls under the characteristic of explainability because it involves clarifying how an AI model arrives at its conclusions, which helps build trust in the system and meet regulatory and ethical requirements.

ISTQB CT-AI Syllabus (Section 2.7: Transparency, Interpretability, and Explainability)

"Explainability is considered to be the ease with which users can determine how the AI-based system comes up with a particular result"​.

"Most users are presented with AI-based systems as ‘black boxes’ and have little awareness of how these systems arrive at their results. This ignorance may even apply to the data scientists who built the systems. Occasionally, users may not even be aware they are interacting with an AI-based system"​.

ISTQB CT-AI Syllabus (Section 8.6: Testing the Transparency, Interpretability, and Explainability of AI-based Systems)

"Testing the explainability of AI-based systems involves verifying whether users can understand and validate AI-generated decisions. This ensures that AI systems remain accountable and do not make incomprehensible or biased decisions"​.

Contrast with Other Options:

Autonomy (B): Autonomy relates to an AI system's ability to operate independently without human oversight​. While decision-making is a key function of autonomy, the focus here is on demonstrating the reasoning behind decisions, which falls under explainability rather than autonomy.

Self-learning (C): Self-learning systems adapt based on previous data and experiences, which is different from making decisions understandable to humans​.

Non-determinism (D): AI-based systems are often probabilistic and non-deterministic, meaning they do not always produce the same output for the same input. This can make testing and validation more challenging, but it does not relate to explaining the decision-making process​.

Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Since the question explicitly asks about the characteristic under which decision-making falls when being demonstrated before deployment,explainability is the correct choicebecause it ensures that AI decisions are transparent, understandable, and accountable to stakeholders​.

AI systems, particularly machine learning models, need to exhibit adaptability and flexibility to handle slight variations in input data without requiring retraining. The ISTQB CT-AI syllabus outlines adaptability as a crucial feature of AI systems, especially when the system is exposed to variations in its operational environment​.

Option A:“It has been trained to recognize cats and is given an image of a dog.”

This scenario introduces an entirely new class (dogs), which is outside the AI system’s expected scope. If the AI was only trained to recognize cats, it would not be expected to recognize dogs correctly without retraining. This does not demonstrate adaptability as expected from an AI system.

Option B:“It has been trained to recognize human faces at a particular resolution and it is given a human face image captured with a higher resolution.”

This is an example of an AI system encountering a variation of its training data rather than entirely new data. Most AI-based image processing models can adapt to different resolutions by applying downsampling or other pre-processing techniques. Since the data remains within the domain of human faces, the model should be able to process the higher-resolution image without significant issues.

Option C:“It has been trained to analyze mathematical models and is given a set of landscape pictures to classify.”

This represents a complete shift in the data type from structured numerical data to unstructured image data. The AI system is unlikely to adapt effectively, as it has not been trained on image classification tasks.

Option D:“It has been trained to analyze customer buying trend data and is given information on supplier cost data.”

This introduces a significant domain shift. Customer buying trends focus on consumer behavior, while supplier cost data relates to pricing structures and logistics. The AI system would likely require retraining to process the new data meaningfully.

Adaptability Requirements:The syllabus discusses that AI-based systems must be able to adapt to changes in their operational environment and constraints, including minor variations in input quality (such as resolution changes)​.

Autonomous Learning & Evolution:AI systems are expected to improve and handle evolving inputs based on prior experience​.

Challenges in Testing Self-Learning Systems:AI systems should be tested to ensure they function correctly when encountering new but related data, such as different resolutions of the same object​.

Analysis of the Answer Options:ISTQB CT-AI Syllabus References:Thus,option Bis the best choice as it aligns with the adaptability characteristics expected from AI-based systems.

 Challenges for Acquiring Test Data in ML Systems:Compliance needs, the changing nature of data over time, and sourcing data from public sources are significant challenges. Data being generated quickly is generally not a challenge; it can actually be beneficial as it provides more data for training and testing.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Data Preparation and Data Quality Issues​​​​.