ISTQB Specialist (CT-PT) Mock Exam #3 — Questions & Answers

Testing the ability of a system to handle increasing levels of anticipated, realistic load over a defined period of time.

Testing whether the system can be expanded, for instance by adding processing nodes or memory, to meet future performance efficiency requirements, and identifying the point at which further expansion stops helping.

Testing the stability of a system over an extended period of continuous execution at a load representative of normal operation, since defects that accumulate slowly only surface after prolonged running.

Performance testing may begin once functional testing has been completed and the system is feature-complete, since partial systems cannot yield meaningful measurements.

Performance testing is primarily a production activity, so measurements taken before release count as indicative background rather than as results.

A performance test has passed when resource utilisation on every server node stays below 75% for the whole run.

Capacity testing of the matchmaking service

Spike testing

Endurance testing

Capacity testing of the current configuration

Load testing

Stress testing

Load generation through captured communication protocols

Load generation through the application programming interface

Load generation through the user interface

Degradation of response time over the duration of operation, which a long-running system develops through memory leaks, fragmentation of the disk, uncontrolled growth of the database tables, and caches that are not reclaimed.

Slow response under all levels of load, including light load, which inefficient algorithms and unoptimised database queries produce even when resource pools are generously sized.

Slow response under moderate to heavy load caused by saturation of a resource such as processor, memory or network bandwidth.

Processor and memory utilisation on the application server during the peak period of the run, sampled per server alongside disk and network counters.

Customer orders completed per hour during the business peak.

The 90th percentile response time of the login transaction measured across the whole steady-state period of the run.

Their primary function is to generate the transaction load for a performance test, with the measurement of resource consumption as a secondary role.

They are the usual source for identifying deadlocks, database exceptions and transaction time-outs, because a monitoring agent attached to the database exposes each such condition as a dedicated counter.

They should be configured to collect every counter available at the highest sampling frequency, so that no anomaly can escape observation during the run.

Since no resource is close to saturation, the measured elapsed time represents the best that the current architecture can deliver, and because further tuning would have nothing left to optimise, the correct response is to renegotiate the acceptance criterion with the business and to record the architecture itself as the binding constraint.

The combination of low utilisation and a long elapsed time indicates that the monitoring tool has produced faulty figures, because counters sampled at a coarse interval under-report short bursts of activity, so the run should be repeated with a different tool before any conclusion is drawn.

Utilisation below 75% on all servers is the accepted pass threshold for a performance test, so the consolidation step meets its objectives and the elapsed time is a business planning matter rather than a performance testing finding.

New measurements cannot be gathered at all until a baseline has been recorded, because performance test tools derive transaction response times relative to a stored baseline run.

The missing baseline affects only the presentation of the results in the management report, since the measurements of the new run remain directly comparable with those of any later run of the same test on the same environment.

The team should compare the new results against published industry averages for document rendering services, which serve as the baseline when local history is missing.

Test planning

Test implementation

Test execution

A modular script can be recorded once and replayed unchanged by every virtual user, because the module boundary isolates the user-specific data from the script logic itself.

Modular test cases take the place of operational profiles, because each module already carries the proportion in which it is to be executed.

Because each module is a self-contained unit, the measurements it produces are independent of the other modules running in the same scenario at the time.

The risk is concentrated in the client device, because a multi-tier architecture moves a substantial part of the routing and formatting work away from the server tiers and onto client software, leaving the tiers lightly loaded.

Because every tier can be scaled independently of the others, the performance risk reduces to choosing the right number of instances for each tier, and identifying which particular tier is responsible for the elapsed time is not part of the exercise.

The dominant risk is that all components share one set of physical resources, so unrelated processes on the same machine compete for the same processor, memory and disk.

The principal risk lies in the speed and congestion of the public network segments between the components, which the project cannot control or reliably reproduce in a test.

The principal risk is uneven load balancing, because incoming requests may be distributed unequally across the available processing units, so that one of them saturates while the others stay idle and the imbalance disappears in aggregated results.

The principal risk is that additional resources are provisioned automatically during the run, changing the configuration under measurement and making the results hard to reproduce from one execution to the next.

It establishes the production capacity of the system on every merge, because figures from a scaled-down environment scale in proportion to the hardware, so a full-scale performance test is needed only when the architecture changes.

Because the environment is scaled down and the run lasts only ten minutes, the check yields no usable performance information and serves purely as a smoke test confirming that the build pipeline and the test scripts themselves still work.

It moves performance risk analysis out of the iteration, since any regression that matters to stakeholders surfaces as a breach of the configured 10% threshold on that operation.

The field reports point to a server-side capacity shortfall that the laboratory test did not reach, because the test ran at the target load rather than above it, so the next step is to repeat the test at a substantially higher load against the same corporate-network configuration and look for the point at which the server finally gives way under sustained overload.

Since the server-side times agree with the test, the field reports describe a functional defect in the mobile client rather than a performance problem; the retry logic in the client is the likely cause and the item belongs in the defect backlog as a client software bug.

The discrepancy shows that the 2.0 s acceptance criterion was written against the wrong metric: the elapsed time a courier sees includes network segments outside the team's control, so the criterion should be restated as a limit on server-side processing time, which can be measured reliably on every run.

The consolidation service is under-provisioned for the volume it receives, since a queue holding 2.4 million messages is far beyond anything it was benchmarked against, so adding processor capacity to that service is the change most likely to restore the benchmarked rate of 1 400 records per second and bring the whole month-end close back inside the 90-minute window required by the business.

The feeder systems are the cause, because the queue would not have grown if they had published their records gradually across the window instead of within the first 20 minutes, so their export targets should be tightened and staggered each night to smooth the flow into the consolidation service.

Since every measured component target was satisfied, the 90-minute end-to-end requirement is unachievable with this architecture, because an end-to-end figure cannot be better than the sum of the component targets plus the transfer time between them, so the appropriate response is to renegotiate the requirement with the business before further tuning is commissioned.

The first is a technical objective because it states a measurable number of seconds that the test tool can verify directly, and the second is a user-based objective because the sustained assertion rate decides how many citizens the service can serve at once during the peak.

Both are technical objectives, since both are measured by the performance test tool; user-based objectives are recorded only in the service level agreement and lie outside the scope of the performance test plan altogether.

The first is an acceptance criterion and the second a test objective, so they belong to different documents: acceptance criteria are owned by the business and recorded in the contract, while test objectives are owned by the test team, and for that reason no traceability between the two is maintained.

The executive summary of the measurements obtained during the run, written in non-technical language for the business stakeholders who commissioned it.

The defect reports raised while the scenarios were executed, with their severity, priority and current status.

The recorded protocol scripts themselves, embedded in full so that the plan is a self-contained deliverable a reviewer can assess without consulting other artefacts.

The interface-driven workstations add load to the protocol-level test, and because a browser session consumes far less generator memory than a protocol session does, this is the least expensive way of pushing the total number of virtual users past the limit imposed by the tool licence.

They provide a second independent measurement of the same server processing time, and because two instruments measuring one quantity are more accurate than one, the two figures are averaged for each transaction so that the random measurement error of the protocol-level tool is reduced before the results are reported.

They stand in for client-side resource monitoring, because an interface-driven script reports the processor and memory consumption of the browser alongside its timings, so no separate monitoring agent has to be installed on the workstations.

The 304 responses are redirections that each add an extra round trip to the server, and the 404 responses show requests the server declined to serve because it was shedding load at that moment.

Both codes belong to the ranges reserved for server-side errors raised under load, so the correct treatment is to add the 304 and 404 counts together, express the total as a share of all responses, and report that share as the failure rate of the run alongside the response-time and throughput statistics for the steady state.

The 304 responses can be counted as equivalent to 200 responses when reporting throughput in bits per second, since both indicate a resource successfully delivered, and the 404 responses indicate that the virtual user lacked authorisation.

The outer timer should equal the sum of the inner timers, because a timer in a script reports only the time during which the system under test was busy, so a difference of this size shows that the inner timers were placed incorrectly and that the script must be re-instrumented before the figures are compared with the acceptance criteria.

The outer timer measures elapsed time on the server while the inner timers measure the time observed at the client, so the 3.2 s difference between them is the network transfer time accumulated across the whole operation and should be reported separately as a network cost.

Performance test tools subtract the configured think time from every timer before reporting it, so no timer in the script can contain a pause; the 3.2 s difference must therefore represent server processing performed outside the three instrumented steps, and the script should be extended with further inner timers until the outer figure is fully accounted for.

The heap size is tuned so that each load generator can host more virtual users within the memory available to it, and the network emulation device is installed between the generators and the system under test in order to cap the bandwidth that the run consumes on the shared corporate link, keeping the generators from saturating it.

Both steps are optimisations of the test infrastructure: they affect how quickly and how conveniently a run can be executed and how easily it can be repeated by another member of the team, but they leave the behaviour of the system under test, and therefore the measurements, entirely unchanged.

Network emulation is a substitute for generating load from several geographic points of presence, since adding latency to a single path reproduces the effect of distant origins, and aligning the heap size with production is what makes extrapolation from a scaled-down environment a linear calculation, so the two steps together allow production figures to be derived arithmetically from the test results.

Yes. They are one quantity expressed in two units, since each completed transaction carries a fixed amount of data across the network, so a performance objective may be written in either unit interchangeably and converted from one to the other as soon as a different audience needs the report.

No. The transactions-per-second figure describes the system under test while the megabits-per-second figure describes the throughput achieved by the load generators, so the two are measured on opposite sides of the test and should not be placed side by side in one report.

No. Transactions per second is a business environment metric while megabits per second is an operational environment metric, so only the first can carry a requirement and the second is purely diagnostic information.

Because cloud generators are provisioned on demand, their own resource consumption falls outside the scope of the test, since the provider sizes each instance to the load it is asked to produce and replaces any instance that saturates during the run.

A software-as-a-service load testing platform gives the tester direct operating-system level access to each generator, which an internally hosted laboratory cannot offer at a comparable cost.

It makes results more reproducible than an internal laboratory, because the provider allocates identical hardware and identical network conditions to every run.

Because the load generation tool derives each transaction response time from the difference between the server clock and the client clock, so any offset between them would make every reported response time wrong by that offset.

Because the monitoring tools write into a shared result repository that is indexed by timestamp, and samples carrying colliding timestamps would overwrite one another and be lost from the result set.

Because the virtual users read the pacing and think times they apply between steps from the clock of the machine they run on, an offset between generators makes some users iterate faster than others, which changes the throughput the scenario actually achieves and invalidates the load profile.

Each script is executed by only one virtual user at a time, so that the measurements it produces stay free of interference from the other scripts running in the same scenario.

The scenario is recorded afresh as one combined script covering every operation, which then supersedes the individual scripts and reduces the maintenance effort.

The scenario fixes the order in which the scripts execute, because the operations must form a strict sequence for the measurements to be valid.

About 625 requests, calculated by dividing the arrival rate of 250 per second by the average time in the system of 0.4 s, since concurrency in an open system is the arrival rate taken per unit of service time, so a shorter service time implies a larger population.

250 requests, because in steady state the number of requests present inside an open system equals the arrival rate expressed per second, the average time in the system affecting only the latency of the response each request receives rather than how many requests are resident in it at once.

It cannot be determined from these figures, because concurrency is a property of the user population rather than of the arrival rate, and computing it requires both the think time applied between iterations and the number of active users, neither of which has been stated for this authentication service.

Production capacity is 3 450 messages per second, since the two environments are configured identically and the number of application nodes is the only difference between them, which makes the relationship linear.

Production capacity is 1 150 messages per second, because a throughput figure measured in a test environment is the only figure that may be reported for the system, irrespective of how many application nodes are deployed in production.

No numerical statement about production may be derived from a scaled-down environment, because the extrapolation would rest on assumptions that cannot be verified, so the measurement is useful only as a baseline for comparing successive builds of the platform with one another in the same test configuration.

45 megabits per second, obtained by multiplying 50 000 by 900 and dividing the product by one million to reach megabits, and the link is adequate with a very large margin left over for the traffic that already uses it.

2 880 megabits per second, and the link is inadequate, because the scoreboard traffic alone exceeds the total capacity of the link before the existing 40% is even taken into account.

360 megabits per second, and the link is inadequate, because the 40% of its capacity already consumed by other traffic leaves only 400 megabits per second free for the scoreboard traffic.

15 120 orders per hour, which falls short of the requirement by 2 880 orders per hour, or 19% of the hourly rate the platform actually achieved.

15 120 orders per hour, which satisfies the requirement once the 40-minute steady state is scaled up to a full hour by a factor of 1.5, giving 22 680 orders per hour.

252 orders per hour, a shortfall of about 98.6%, found by multiplying 4.2 per second by 60.

The sawtooth shape of the heap graph shows memory being allocated and not released, so the response-time spikes are the early stage of a memory leak that a longer endurance run would turn into an outright failure.

The spikes indicate intermittent saturation of the processor on the application server: the reported 61% to 64% is an average over each sampling interval, so bursts of requests arriving close together can still drive utilisation to its limit for a few seconds at a time without raising that average noticeably.

Since both the average response time and the 90th percentile stay well inside the 2.00 s criterion and no transaction failed, the maxima are statistical outliers of no practical consequence and the transaction meets its acceptance criteria.

The pattern is the classic signature of a memory leak in the search component: response times rise steadily while the applied load is held constant, and the fall in throughput from 210 to 190 transactions per second is the second symptom, as the component spends a growing share of each iteration in garbage collection.

The drift shows a resource leak in the database connection pool: connections are being taken from the pool and not returned, so fewer remain available as the run proceeds, which explains both the response time rising from 0.42 s to 1.18 s and the throughput falling from 210 to 190 transactions per second while the applied load stayed at 700 virtual users.

The run had not reached steady state until its final 20 minutes, because a four-hour test needs several hours before its caches and connection pools settle, so only that last window represents the system and only those figures should be reported as the result.

Because the stub isolates the matchmaking service from its dependency, the figures measure that service's own performance correctly, and the production response time is obtained by adding the entitlement service's mean of 180 ms to each measurement, which gives 0.52 s on average and 0.64 s at the 90th percentile, both inside the criterion.

The stub makes the test pessimistic rather than optimistic. A fixed canned response is returned for every call, so it cannot benefit from the caching that the real entitlement service performs for repeated lookups of the same player, and the production figures of 180 ms mean and 320 ms at the 90th percentile are themselves averages over cold lookups that a warm cache would shorten considerably.

The results are valid as they stand, since service virtualization is the accepted approach when an external dependency is unavailable, and the acceptance criterion applies to the matchmaking service rather than to the provider's entitlement service, whose timing is the provider's own responsibility.

Since the cluster-wide average of 1.18 s is inside the 1.50 s criterion and no transaction failed, the run passes and the per-node breakdown is informational detail that does not affect the verdict.

Node-1 must be defective hardware and should be taken out of the cluster and replaced, because six nodes built to one specification and sitting behind the same load balancer would be expected to report comparable throughput and comparable response times, so a node answering in 1.94 s where its peers answer in 0.87 s can only be explained by a fault in the machine itself.

The differences between the nodes lie within the normal variation of a six-node cluster, so the single actionable finding is the 93% utilisation on node-1, which should be addressed by adding processor capacity to that node.

Build B is the better choice, because its 90th percentile of 1.18 s is marginally lower than build A's 1.20 s while the two means are identical, and the 99th percentile describes rare outliers that should be left out of acceptance criteria as unrepresentative of ordinary use.

The identical means of 0.92 s demonstrate that the two builds perform equivalently under this load profile, and since both runs used the same data set, the same load profile and recorded no failed transactions, the gap at the 99th percentile is measurement noise from the test environment that would not be expected to reappear if the two runs were repeated.

The criterion should be rewritten around the maximum response time observed in each run, since the maximum is the single figure that captures the slowest experience any user had, it is simpler for stakeholders to agree on than a percentile, and build B's maximum of 9.60 s against build A's 1.81 s separates the two builds at a glance.

Both sets of figures are valid views of the same run, and because the full-run aggregate is computed over a larger number of samples it is the statistically more reliable of the two and should be the one reported.

The difference shows the system degrading as the run progresses: resources accumulate over the 90 minutes, so the later the sample the slower it is, and the steady-state subset is worse simply because it is weighted towards the second half of the run, where the slowest transactions were recorded.

The steady-state figures are inflated because excluding the ramp-up discards the transient states, and transient states are where mass-login problems appear, so the full-run aggregate remains the correct basis for judging acceptance.

The console produces the load for those transactions that require synchronisation between virtual users, such as rendezvous points, while the generators produce the load for the remaining independent transactions and report their individual timings back to the console once the run has finished.

The console monitors the resource consumption of the system under test and raises an alert when a threshold is crossed, while each generator monitors only its own resource consumption and reports it in a result set kept separate from the transaction timings it recorded.

The console holds the licences and issues them to the generators at run time, which is its sole function, with the scenario configured separately on each generator machine and the separate result sets merged by hand afterwards.

How many concurrent virtual users the tool can drive, and how its licences count those users and the machines they run on.

How much technical knowledge the tool requires of the people operating it, given that insufficiently skilled testers can produce incorrect results with it.

Whether the operating system counters the tool gathers can be related to the transaction response times it records on a shared timeline, and whether the set of counters can be supplemented from other sources.

Scalability. The tool's ability to drive 3 000 virtual users from the machines available is what governs whether the degradation can be reproduced for a second and better instrumented attempt.

Compatibility. The two data sets are exported in different formats with timestamps rounded differently, and support for data formats and for the exchange of results with other components is what the compatibility factor is concerned with, so the shortcoming should have been caught when the formats were assessed.

Understandability. The team failed because it lacked the technical knowledge needed to interpret operating system counters such as disk queue length and to relate their movements to the behaviour of the application under load.