Manual Testing, Penetration Testing, and Red Teaming

The invoice service has reached an encouraging point. Code review found the missing tenant boundary. Automated tests now reject cross-tenant invoice IDs, fail closed when policy is unavailable, and stop sensitive payloads from reaching the log. The release pipeline is green.

One claim remains largely untested: can a billing editor make only the corrections the business intended, across every path that performs the work?

A scanner cannot explore that claim unaided. It does not know that a settled invoice should be immutable, that an old bulk-import route reaches the same worker as the new correction screen, or that two individually permitted actions can become harmful when chained. A person can follow those seams. The value of manual security testing lies in that directed judgment, not in performing automated checks by hand.

The work is complete only when a bounded question has become a verified change: the exploit path no longer works, the weakness has regression protection, defenders can see the behavior that matters, and an owner has made the residual-risk decision.

Choose the smallest exercise that can answer the question

“Manual test” names a method, not a useful scope. Start with the uncertainty.

The invoice question needs focused application and API testing. The tester will reason about roles, tenant boundaries, invoice states, retries, and alternate entry points. If the uncertainty instead concerned cloud administration, the work would follow identities through roles, account boundaries, storage, networks, keys, and management-plane logs. An infrastructure assessment would concentrate on reachable services, host controls, identity paths, and segmentation. Mobile testing would include device storage, platform permissions, transport, and the boundary between the client and its APIs. AI abuse testing would examine how hostile instructions or retrieved content can influence data access, tool use, output handling, and resource consumption.

A penetration test is broader: it asks whether an agreed target can be compromised under defined rules, then follows the reachable impact far enough to establish exploitability. It still needs a business question. “Find as much as possible in two weeks” encourages a tour of familiar vulnerabilities; “determine whether an internet user can obtain another tenant’s financial data or gain a privileged billing action” creates a defensible investigation.

Red and purple team exercises ask a different question. A red team attempts an authorized adversary path while limiting what defenders know, so the organization can observe prevention, detection, decision-making, and response under realistic uncertainty. A purple team makes attacker and defender collaboration explicit: execute a known behavior, inspect the telemetry, tune the control, and run it again. Neither is a grander name for an application assessment. If the invoice team needs a root cause in one workflow, give it a focused test rather than an enterprise exercise.

Turn the invoice claim into a scope

The test lead writes the objective as a proposition that can fail:

A billing editor can correct a draft invoice only within the editor’s tenant. The editor cannot correct a settled invoice through the user interface, direct API, bulk import, retry, or worker path. Every denied or completed attempt produces enough evidence to reconstruct the actor, invoice, tenant, entry point, policy decision, and resulting state.

That statement determines the systems in scope: the web application, correction API, legacy bulk-import route, queue, worker, policy service, invoice store, and security event stream. It also exposes a deliberate boundary. Payment settlement itself is out of scope; the test may use only seeded invoices with a fake payment provider. A newly discovered path into real settlement stops the exercise and triggers escalation rather than becoming permission to explore.

The scope record should fit on one page. Name:

• the objective and the security assumptions it challenges;
• exact domains, APIs, builds, environments, accounts, source ranges, tenants, and data sets in scope;
• the start, end, blackout periods, on-call coverage, and communication channel;
• the permitted techniques, rate limits, privilege changes, and any approved chaining between systems;
• prohibited actions, including destructive state changes, persistence, unbounded load, access to real customer data, and contact with people unless separately authorized;
• the evidence store, who may access it, required redaction, retention, and deletion date;
• the safety owner, rollback owner, stop conditions, emergency contact, and person authorized to resume.

These are the rules of engagement. They are operational controls, not contractual decoration. For the invoice test, a burst of requests that threatens the shared test environment, evidence of production credentials, or any effect outside the seeded tenants causes an immediate stop. The tester preserves the minimum evidence, contacts the named safety owner, and waits for an explicit decision. The scope should also say whether a tester may continue from one weakness into another. Without that permission, proving access to a forbidden invoice does not authorize changing it or using its contents elsewhere.

Prepare the test world to resist easy answers

A single account and three identical invoices can make a broken system look sound. The useful test world contains tenant A and tenant B, ordinary and privileged billing roles, draft and settled invoices, a disabled account, duplicate correction requests, and a worker retry after an uncertain response. These states let the tester vary one boundary at a time and recognize an outcome that should not exist.

Before testing, the engineering owner provides the current architecture and threat model, API and worker contracts, authentication flow, data classification, known exceptions, previous findings, and expected event schema. The tester receives purpose-built accounts rather than borrowed employee credentials. Operations confirms rollback or restore readiness for any mutable state and verifies that the emergency channel is attended.

Logging deserves a check before the exercise. Send one harmless denied request and follow its request identifier across the edge, API, policy decision, queue, and worker. If the records cannot distinguish the acting user from the worker’s service identity, the team has learned something important—but it has not prepared a reliable observation surface. Repair or explicitly record that limitation before attempting a chain whose outcome depends on attribution.

Preparation should not disclose the answer. Engineers can explain intended boundaries and safe handling without pointing to the suspected defect. Conversely, withholding architecture, test accounts, or safety contacts to make the exercise feel more realistic often measures environmental confusion rather than the control under study.

Follow authority through every path

The tester first establishes the ordinary behavior: an editor in tenant A corrects a draft invoice in tenant A, and the API, queue, worker, database, and event stream agree about the result. Then one fact changes at a time. Use a tenant B invoice. Use a settled invoice. Submit through the old bulk route. Repeat the request. Disable the actor after enqueue but before execution. Force the permitted test double to return an uncertain result and observe the retry.

The interesting failure appears in the alternate path. The interactive API checks the invoice state before enqueueing a correction, but the bulk route accepts a row for a settled invoice. The worker trusts that upstream validation occurred and applies the change under its service identity. The policy log records a permitted worker action but loses the initiating user and entry point. No single component performs an obviously absurd operation: the bulk route accepts a syntactically valid record, and the worker consumes an authorized message. The chain violates the business rule because the authority and state invariant dissolved between them.

Stop at the evidence needed to establish the consequence. Record the seeded invoice, actor, tenant, route, request and job identifiers, state before and after, relevant policy decisions, and timestamps. Do not collect unrelated records or turn a proven control failure into a larger impact merely to improve the finding’s severity.

This is where human testing earns its cost. The tester did not guess random inputs until an error appeared. They held one security claim against several representations of the same business action and followed authority through time.

Write a finding an engineer can resolve

“Authorization bypass,” accompanied by screenshots, leaves most of the engineering work undone. The finding should name the failed invariant and the affected path:

Bulk correction bypasses settled-invoice immutability. A tenant A billing editor can submit a settled tenant A invoice through the legacy bulk route. The route omits the state check enforced by the interactive API, and the worker assumes its caller already validated the transition. The correction changes financial state under the worker identity, while the audit event omits the initiating user and entry point.

The report then records:

• Impact and scope: which invoice states, roles, tenants, environments, and business or data consequences are confirmed; what was not tested.
• Safe reproduction: the minimum sequence an owner can repeat with seeded data, including identifiers and expected versus observed state.
• Evidence: controlled requests, responses, traces, events, configuration, and before-and-after records, with sensitive material redacted or referenced from the approved evidence store.
• Root cause: the business invariant exists in one entry point rather than at the authoritative state transition, while the message contract drops actor context.
• Remediation: enforce permitted transitions transactionally at the authoritative write, carry verified actor and entry-point context through the job, reject stale or duplicate work, and remove or constrain the legacy route.
• Detection opportunity: alert or review on rejected settled-invoice corrections, bulk corrections outside expected volumes, and jobs whose initiating actor context is absent.
• Ownership and decision: engineering owner, detection owner, target date, retest owner, temporary control if any, and the authorized risk decision.

Severity follows the demonstrated path and consequence. Ease of exploitation, exposed roles, financial integrity, scale, detectability, and recovery all influence urgency. A long chain in an isolated lab is not automatically more severe than a one-request violation against sensitive production state. Separate what the test proved from plausible extensions that were outside its rules.

Repair the invariant, then replay the path

Disabling bulk import may be a sensible immediate containment, but it does not by itself repair the shared design. The durable fix moves the invoice-state transition rule to the authoritative write so every caller encounters it. The job carries an authenticated actor reference and original entry point; the worker reauthorizes time-sensitive work and treats duplicate or stale commands explicitly. Security events bind the actor, tenant, invoice, transition, source route, policy result, job, and deployed version without recording invoice contents.

Validation begins with the original reproduction. The settled invoice remains unchanged through the bulk path, and the denial event preserves enough context for investigation. Then widen the check: the interactive path still permits an authorized draft correction; tenant B remains inaccessible; duplicate jobs cannot apply twice; a disabled actor cannot rely on an old queued decision; and retries after uncertain outcomes converge on one recorded result.

Promote the stable cases into automated regression tests. Keep the part that requires exploratory judgment in the next manual scope. A regression test protects a known path; it does not make future human investigation unnecessary.

For a high-impact finding, the original tester or an independent security engineer should perform the retest. Closure requires more than a changed status in a ticket: the exploit no longer works, the root cause is addressed across equivalent paths, regression evidence is tied to the fix, telemetry has been verified, and any remaining exposure has an owner and explicit risk decision.

Use attacker behavior to improve defense

Suppose the invoice path is one step in a broader exercise: a compromised billing account uses the legacy route, creates an unusual volume of corrections, and attempts to hide among normal worker traffic. A red team may test whether the organization notices and coordinates a response under the agreed limits. The finding is not only the missing state check. It may include weak identity signals, indistinguishable service actions, an alert that nobody owns, slow containment, or a recovery procedure that cannot identify affected invoices.

A purple team can work the same behavior openly. Before execution, attacker and defender name the hypothesis: a burst of settled-invoice correction attempts should produce attributable denial events and reach the billing-security on-call within the agreed window. They approve the accounts, environment, test window, and exact safe actions. During execution they mark which events appear, which query joins them, whether the alert fires, who receives it, and which response step follows. After tuning the event, rule, routing, or runbook, they repeat the behavior. The evidence is the second observed result, not a meeting note that everyone now understands the gap.

Surprise is useful only when readiness under uncertainty is itself the subject. Most manual assurance improves faster when builders, testers, and defenders share the question, observe the path, and shorten the repair loop.

Carry the result into the release decision

Manual testing fails quietly when its report becomes a terminal artifact. Vague objectives yield generic findings. Unsafe or impoverished test accounts conceal important paths. Screenshots substitute for reproducible evidence. Root causes become a list of isolated bugs. Detection is mentioned but never exercised. A fix is accepted without replaying the path.

The invoice test is finished when its original proposition can be answered precisely. Draft corrections work for the right actor and tenant. Settled corrections fail at the authoritative boundary regardless of entry point. The event trail retains the initiating authority across asynchronous work. Known cases have regression protection, the consequential behavior is detectable, and the remaining uncertainty is recorded.

That evidence now belongs with the threat model, automated results, open findings, operational controls, and exception records. The next question is no longer whether the test produced a report. It is whether the whole release is safe enough to enter production—and who is authorized to accept what remains unknown.