Procedural Macros: Token Streams, Spans, and Diagnostics

Where should this error point?

#[derive(RouteSpec)]
#[route(method = "PATCH", path = "accounts", secret = true)]
struct Accounts;

There are three different defects. secret is not part of the attribute language. PATCH is outside the service’s supported method set. The path lacks its leading slash. A macro that returns one compile_error! at the derive name has technically rejected the program and still failed its user. A maintainable derive reports independent failures at the tokens the user can change.

That diagnostic decision determines the architecture. The entry point must preserve tokens and spans. Parsing must produce structure without silently discarding unknown input. Validation must accumulate domain errors. Generation must run only on valid input and attach user-owned spans to relevant output. The proc-macro crate must remain a thin compile-time adapter around ordinary testable logic.

The central rule is: treat a procedural macro as a compiler-facing boundary, not as a clever function that happens to return tokens. It executes during compilation, consumes a syntax-bearing token stream, and can affect every downstream build that trusts it.

Put the crate boundary in the design

Rust procedural macros live in a crate whose library target has proc-macro = true. Exported derive, attribute, and function-like entry points are public functions at that crate’s root with signatures based on proc_macro::TokenStream. A proc-macro crate is not an ordinary runtime library. Consumers load its compiled macro for the host and execute it while compiling their own crate, including during cross-compilation for a different target.

The lab uses three crates:

macro-contract    ordinary library: Route, RouteSpec, routes!
route-derive      proc-macro library: #[derive(RouteSpec)]
macro-demo        downstream consumer and behavioral tests

Keeping the trait in macro-contract avoids asking runtime users to depend on implementation details of the compiler plugin. The derive crate depends on syntax tooling and emits an implementation of ::macro_contract::RouteSpec. The consumer opts into both. Some ecosystems re-export the derive from the runtime crate for ergonomics, but the physical host-executed crate boundary still exists and should remain visible in dependency and security review.

Do not put domain behavior, network clients, file discovery, or runtime registration into the macro crate. A custom derive should usually translate declared source intent into ordinary implementations. Hidden runtime side effects make expanded code hard to audit and couple compile-time convenience to operational behavior.

The entry point in the fixture is intentionally small:

#[proc_macro_derive(RouteSpec, attributes(route))]
pub fn derive_route_spec(input: TokenStream) -> TokenStream {
    let input = parse_macro_input!(input as DeriveInput);
    expand(input)
        .unwrap_or_else(syn::Error::into_compile_error)
        .into()
}

It declares the route helper attribute, parses a derive input, delegates, converts structured errors into stable compile errors, and crosses back into the compiler token type. The architecture is inspectable because policy is not buried in this adapter.

Token streams preserve syntax boundaries

TokenStream is a sequence of token trees: identifiers, punctuation, literals, and delimited groups. It is cheaper and more stable as an interface than exposing the compiler’s complete internal abstract syntax tree. Converting the input to a string and parsing the string throws away the main advantages of this boundary: source spans, punctuation spacing, reliable literal handling, and structured groups.

Tokens are not yet the domain model. In the example, the stream contains the struct item, generic parameters, attributes, fields, and their spans. The derive cares about the item name, generics, and route attribute. A syntax parser turns those tokens into a DeriveInput; domain parsing turns nested attribute metadata into method and path literals; validation decides whether their values are allowed.

Keep those stages distinct:

TokenStream
  -> Rust syntax (`DeriveInput`)
  -> macro input (`RouteArgs`)
  -> validated model (`ValidatedRoute`)
  -> generated implementation

The fixture compresses RouteArgs and ValidatedRoute into local variables because its grammar has only two keys. A production macro with defaults, aliases, field roles, or mutually exclusive options should create explicit types. Invalid states then stop flowing into generation, and unit tests can exercise validation without invoking rustc.

Using a parser library is an ecosystem choice, not a language requirement. syn, quote, and proc-macro2 provide a widely used separation between parsing, token construction, and compiler-facing types. They add dependencies and compile work. For an extremely small function-like macro, direct token iteration may be defensible. Hand parsing a complete Rust item is not. Choose the smallest parser surface that preserves correct Rust grammar and useful spans, and pin the exact versions behind reproducible evidence.

Never use input.to_string() as the semantic protocol. A token stream’s display form is intended for representation, not lossless source reproduction. Whitespace can change, joint punctuation can be surprising, and spans do not survive string round trips. If a generated identifier or literal must be created, construct the appropriate token or use quoted syntax with a deliberate span.

Parse completely or reject explicitly

User intent includes unknown input. Silently ignoring an attribute key because the current version does not understand it converts a typo into changed behavior:

#[route(metohd = "GET", path = "/accounts")]

The fixture rejects every key except method and path:

if meta.path.is_ident("method") {
    set_once(&mut method, meta.value()?.parse()?, meta.path.span(), "method")
} else if meta.path.is_ident("path") {
    set_once(&mut path, meta.value()?.parse()?, meta.path.span(), "path")
} else {
    Err(meta.error("unsupported route key; expected `method` or `path`"))
}

It also rejects duplicates. “Last value wins” is dangerous for compile-time policy because a merge conflict or generated attribute can silently override a reviewed value. If repetition is meaningful, represent it as a collection and specify order; otherwise treat the second occurrence as an error at the second key.

Parsing and validation answer different questions. Parsing asks whether method = "PATCH" has the expected syntactic form and literal type. Validation asks whether PATCH is supported. The distinction gives diagnostics a stable taxonomy and makes compatibility decisions explicit. Adding a new accepted method is a domain-language change, not a parser bug fix.

Preserve generics, lifetimes, const parameters, and where clauses from the input. The lab calls split_for_impl() and uses all three returned pieces:

let (impl_generics, type_generics, where_clause) = input.generics.split_for_impl();

quote! {
    impl #impl_generics ::macro_contract::RouteSpec
        for #name #type_generics #where_clause
    {
        const METHOD: &'static str = #method;
        const PATH: &'static str = #path;
    }
}

Dropping a where clause may make a generic derive fail only for downstream users. Inventing broad trait bounds can make an implementation unavailable even when the generated code does not require them. Generated bounds are public API; preserve syntax you do not own and add only obligations justified by emitted operations.

Field-based derives need the same discipline. Preserve field order when order has meaning, accept named, tuple, or unit forms only when the contract defines them, and reject unsupported unions or enum shapes at the relevant keyword or variant. Do not let a parser library’s ability to parse a form imply that the macro semantically supports it.

Spans are part of the user interface

A Span associates a token with source location and expansion context. Stable procedural-macro APIs let authors carry existing spans and apply spans to generated tokens, while some richer diagnostic operations remain toolchain-sensitive. The most durable stable strategy is to emit compile_error! tokens carrying the span of the invalid user input.

The example attaches each domain error to its literal:

if !matches!(method.value().as_str(), "GET" | "POST" | "PUT" | "DELETE") {
    combine(
        &mut errors,
        syn::Error::new(method.span(), "method must be GET, POST, PUT, or DELETE"),
    );
}

if !path.value().starts_with('/') {
    combine(
        &mut errors,
        syn::Error::new(path.span(), "route path must start with `/`"),
    );
}

The underlined text is user-owned and actionable. A missing key has no literal, so the fixture points at the type identifier. That location is a design choice: the derive site or full attribute span could also be reasonable. State a policy and test it. Do not attach every failure to call-site span merely because it is convenient.

Generated syntax also carries spans. Tokens interpolated from parsed input retain their useful association; tokens created by the macro receive spans chosen by the quoting or construction API. Poor choices can make a type error appear inside inscrutable generated code. Where an error follows from a user-supplied type, field, or attribute, use that input token to anchor the generated obligation when the construction API permits.

Spans are not byte offsets that a macro may freely inspect and rewrite across files. Their stable capabilities are intentionally limited. Do not build correctness around source-text recovery, absolute paths, or unstable location APIs. If exact source preservation matters—for example, a formatter—procedural macro expansion is probably the wrong interface.

Accumulate independent errors. The invalid lab input should report the unknown secret, unsupported method, and malformed path in one compiler run. Failing fast is appropriate when later validation depends on a missing parse tree; it is needlessly expensive when three independent attribute values are already available. Combine errors in deterministic source order so output is reproducible.

Error messages should name the violated contract and accepted repair, not internal parser types. “expected LitStr” is less useful than “path must be a string literal.” Avoid embedding exact dependency implementation details in UI snapshots. Assert stable phrases and span locations tightly enough to catch regression while tolerating irrelevant rustc wording changes.

The full path is easier to retain once parsing and diagnosis have been separated in prose. In the figure, follow the upper lane from user syntax to generated implementation, then use the lower lane to check that every rejection returns to a token the caller owns.

Hygiene is limited, so generate explicit paths

Procedural macros are not hygienic in the same way as declarative macros. Their output behaves much like syntax written at the invocation site. Unqualified names can resolve to a caller’s imports or collide with caller items. Generated public names can alter the surrounding namespace.

Use absolute paths for language and standard-library items when the dependency contract permits it:

::core::option::Option
::std::vec::Vec

The lab emits ::macro_contract::RouteSpec. That creates a different problem: downstream users can rename the dependency in Cargo.toml. An absolute spelling is collision-resistant but not rename-aware. Common strategies include re-exporting the derive through the runtime crate and emitting a documented canonical path, accepting an explicit crate-path override, or resolving package renames through additional tooling. Each adds compatibility surface. Test renamed-dependency behavior rather than assuming a leading :: solves it.

Do not inject imports into the caller unless the macro form clearly owns a module. Fully qualify generated references. Choose internal generated identifiers with low collision risk, and keep them inside a const block or private module where possible. If a derive must emit an associated method, that method name is public surface and belongs in the API review.

Preserve user intent around visibility. A derive generally implements a trait for the input type; it should not make private fields public or generate unrelated exported functions. An attribute macro that replaces an item has greater responsibility: it must decide which attributes, documentation, configuration gates, visibility, ABI, and signatures survive. “Parse then quote most of it back” is not a preservation policy.

Generated code should not depend on the caller importing a trait unless requiring that import is explicit API. It should not change panic, allocation, blocking, or I/O behavior invisibly. Inspect the expanded contract as ordinary Rust: which calls occur, which bounds appear, which names exist, and which runtime costs are introduced?

Keep generation boring and deterministic

Once validation produces a legal model, generation should be close to mechanical. A template with a few interpolations is easier to audit than conditional token assembly scattered through validation. Deterministic input should produce deterministic output independent of hash iteration order, wall clock, locale, current directory, or network state.

Sort only when the public semantics define order as irrelevant. Otherwise preserve source order. Stable generation improves incremental builds, diagnostics, review diffs, and reproducibility. Do not embed timestamps, absolute source paths, random identifiers, or environment-derived features in output.

The output should be the smallest ordinary Rust implementation that carries the contract. The route derive emits two associated constants. It does not create a registry, global constructor, hidden allocation, or asynchronous initialization path. A reader can predict runtime behavior from the trait.

Quoting libraries reduce punctuation mistakes, but quoted syntax can still be wrong. Compile downstream examples. Exercise generic, lifetime, const-generic, where-clause, renamed-field, raw-identifier, and configuration cases appropriate to the macro. Unit tests of the validator are not evidence that the emitted tokens type-check.

Snapshotting expanded text is useful for human review when canonicalized, but brittle as the only oracle. Formatting and compiler representation can change without semantic change. Prefer a portfolio:

• unit tests for domain parsing and validation;
• behavioral tests through the public trait or generated API;
• doctests for the advertised invocation;
• compile-fail UI cases for user-facing diagnostics;
• expansion snapshots for a few complex shapes;
• downstream crates covering editions, features, dependency renames, and MSRV.

The lab uses positive behavior, a public declarative doctest, and a standalone compile-fail derive. The UI script asserts all three independent validation messages. A mature macro should use a maintained UI-test harness that also manages normalized stderr and span expectations, but the dependency-free shell probe makes the evidence transparent.

Budget dependencies and compile time

Every proc-macro dependency is built for and executed on the host during compilation. A large syntax stack can dominate clean builds for a small runtime crate. Feature selection matters: enabling a parser library’s complete syntax model is convenient, but a derive that reads only item metadata may need less.

Measure before optimizing. Record clean build time, incremental no-change build, rebuild after changing only an invocation, host artifacts, dependency tree, and peak memory in the environments where developer and CI latency matter. Separate macro execution time from the cost of compiling macro dependencies and type-checking expanded output. Shrinking generated code may help more than micro-optimizing token iteration.

Avoid duplicate syntax stacks across workspace macros by aligning compatible versions where governance permits. Pinning exact versions in a teaching fixture makes evidence reproducible; a public library may choose compatible requirements and rely on its lockfile in CI. Either way, review updates because parser behavior and supported Rust syntax change.

Proc macros also interact with MSRV. The macro crate’s compiler features, its dependencies’ MSRVs, the syntax it accepts, and the syntax it emits must all fit the consumer policy. A macro compiled by a newer host cannot emit newer syntax into a crate promised to an older compiler and call the result compatible. Test the actual minimum toolchain with a locked graph.

Input size and expansion complexity need bounds. A derive across thousands of fields, deeply nested metadata, or recursive generated types can become a compile-time denial of service. Reject unreasonable nesting where the domain has a natural maximum, avoid repeated cloning of large token streams, and report cost regressions like runtime regressions.

Treat proc macros as supply-chain execution

A procedural macro is executable build-time code from a dependency. When a developer runs cargo check, that code runs with the developer’s build-process permissions. It can read files and environment variables, consume CPU and memory, and—unless the environment prevents it—attempt external effects. Rust’s type safety does not sandbox a proc macro.

Apply the build-dependency trust policy:

• minimize direct and transitive dependencies;
• review ownership, releases, source provenance, licenses, and build scripts;
• pin and audit the lockfile used in CI and release;
• isolate builds handling secrets; pass only necessary environment;
• deny network in hermetic builders where feasible;
• monitor unexpected filesystem access, output growth, and build-time regressions;
• treat a compromised macro as a source-compromise event, not merely a bad library call.

Do not read arbitrary project files from a proc macro to create an implicit build protocol. Cargo cannot reason reliably about hidden inputs, incremental invalidation becomes fragile, and cross-compilation paths differ. A build script with declared rerun inputs, checked-in generated source, or external generator may be the better mechanism when the input is not Rust syntax.

Input tokens can also be hostile in an ecosystem that expands third-party generated source. Parsers should avoid pathological recursion and panics. A macro panic produces a compiler error but gives poor diagnostics and can disrupt tooling. Return structured errors for invalid input. Reserve panic for internal invariants that indicate a macro defect, and test fuzzed or adversarial token shapes when the parser is complex.

Generated code can create security defects without unsafe code: authorization checks can be omitted, sensitive values can enter diagnostics, identifiers can collide, feature gates can invert, or runtime registration order can change. Review semantic output and threat boundaries, not only memory safety.

Stable APIs constrain the architecture

Build the core path on stable proc_macro interfaces and stable language syntax. Some richer span locations, diagnostic builders, token expansion operations, and tracking facilities have evolved behind feature gates. An attractive nightly diagnostic is not a valid default for a stable library unless the book and crate explicitly accept nightly coupling.

The stable fallback—spanned compile_error! output—can support excellent errors when parsing retained the right tokens. This is why span preservation must be architectural, not a late formatting step. If a desired warning or suggestion is unavailable on stable, document the limitation and prefer a clear error over conditional compiler internals.

Separate documented guarantees from ecosystem conventions. TokenStream and macro entry signatures are language and standard-library interfaces. syn error combination and quote! interpolation are third-party library behavior. Exact rustc formatting and expansion order details can change. Record compiler, crate versions, definition edition, and target host with diagnostic evidence.

Version the macro input language like any other public API. Tightening validation can break code that previously compiled; accepting new syntax can change interpretation; changing emitted bounds or names can break type checking; improving a span is usually compatible but can affect snapshot tests. Centralize grammar and validation so every compatibility change remains reviewable.

Run a first audit from entry to evidence

Review the lab in the order data moves:

Boundary

Confirm the proc-macro crate contains only compile-time adapter code. Identify every dependency and whether it runs a build script. Record host compiler, edition, MSRV, and locked versions. Verify the runtime trait lives outside the proc-macro crate.

Parse

List every supported input item and helper attribute form. Prove unknown keys, duplicates, malformed literals, and unsupported item forms are rejected. Confirm no semantic decision depends on token-stream string formatting.

Validate

State domain invariants independently of generated code: exactly one method, exactly one absolute path, and a closed method set in this teaching service. Decide whether errors are independent and verify deterministic combination order.

Generate

Inspect the complete implementation. Preserve generics and where clauses. Inventory added bounds, paths, names, visibility, runtime calls, allocation, panic, and initialization. Test the emitted code through the public contract rather than only comparing tokens.

Diagnose

For each invalid fixture, name the user-owned token that should be underlined. Ensure the message gives an accepted repair without exposing parser internals or secrets. Verify exact wording only where the project intentionally treats it as stable.

Operate

Measure clean and incremental cost, expansion size, and dependency graph. Run builds in a restricted environment. Establish owners for compiler/edition upgrades and parser dependency advisories.

For the exercise, extend RouteSpec with an optional integer priority. Define its range, duplicate policy, default, span policy, emitted type, and compatibility impact before changing code. Add one positive generic type, one malformed literal, one out-of-range value, a duplicate, an unknown key, and a renamed-dependency consumer. Then write a short threat review: which compile-time inputs are trusted, what the macro can access, what generated runtime behavior appears, and how a compromised parser dependency would be contained.

Reject two tempting repairs. Do not parse priority from TokenStream::to_string() to avoid learning nested metadata. Do not generate a runtime parse().unwrap() so invalid configuration becomes a startup panic. Both move a compile-time language obligation into a weaker and less diagnosable boundary.

A complete failure record for RouteSpec

Run the invalid fixture as if it were a downstream report, not a unit test owned by the macro author:

#[derive(RouteSpec)]
#[route(method = "PATCH", path = "accounts", secret = true)]
struct Accounts;

The parser first constructs a valid DeriveInput; the Rust item itself is well formed. Nested metadata parsing accepts the route(...) container, recognizes two key/value pairs, and rejects secret at the key’s span. That error is retained. Validation can still inspect the parsed method and path, so it adds one error at "PATCH" and another at "accounts". into_compile_error() emits multiple compile-error fragments. rustc presents all three in the same run.

This outcome proves more than “bad input fails.” It proves unknown keys are not ignored, semantic validation continues after an independent metadata error, messages retain distinct spans, and order is deterministic. The UI check searches for three contract phrases rather than the entire stderr document. The exact border characters, note ordering, and “originates in derive macro” note belong to the compiler presentation and can change. The violated rules and source associations belong to the macro API.

Now alter the fixture to omit path while retaining an unknown key. Missing required data prevents creation of the validated model, but the unknown-key error should not disappear. The fixture’s merge function combines accumulated parsing errors with the missing-key error. There is no sensible path-value validation because no path token exists. That is principled partial progress: report everything supported by available structure, not speculative cascades.

A production implementation can make this clearer with explicit types:

struct ParsedRoute {
    method: Option<LitStr>,
    path: Option<LitStr>,
    errors: Option<syn::Error>,
}

struct ValidatedRoute {
    method: LitStr,
    path: LitStr,
}

ParsedRoute::validate consumes the partial representation and returns either a complete model or combined errors. Generation accepts only ValidatedRoute. The type boundary prevents a future branch from quoting a missing value or skipping a newly added invariant. For more complex derives, use domain types for method and normalized path rather than carrying arbitrary strings through generation.

Consider how four changes flow through the architecture.

First, supporting PATCH changes only domain validation and tests. The parser already accepts a string literal and generation treats the validated value mechanically. If PATCH is added without changing runtime service capability, the compile-time and runtime contracts diverge; the ordinary trait library must evolve in the same review.

Second, accepting method = GET without quotes changes the public grammar. Parsing must accept either an identifier or literal and normalize both into one method domain type. Diagnostics and migration policy must decide whether two spellings remain permanently supported. Generation should not care which spelling the caller used unless preserving it has semantic value.

Third, allowing the crate dependency to be renamed does not belong in validation. It changes generated path resolution. One design adds #[route(crate = my_contract)] and parses a Rust path, not a string. Another uses a helper crate to discover the dependency name from Cargo metadata conventions. The explicit path is more verbose and deterministic; discovery is more ergonomic and adds compile-time dependencies and failure modes. Both require a downstream test with a renamed dependency.

Fourth, emitting a registration function changes runtime behavior. It adds a name, perhaps a global side effect, initialization order, panic behavior, and possibly allocation. That is not a generation refactor. It requires an API and operations review, and it may reveal that derive is the wrong macro form. A derive should describe behavior naturally tied to a type; service-wide registry construction may belong in a function-like macro, link-time inventory mechanism, build-generated table, or explicit runtime builder.

Record the evidence as a matrix:

The residual-risk column prevents a green fixture from becoming a universal claim. This derive does not detect duplicate paths across types because derives expand independently and do not own a global source registry. It does not promise canonical dependency renaming. Its compiler-time measurement on one host is not a CI fleet budget. The correct response is to scope the API and schedule evidence, not add hidden global state to expansion.

An implementation review ends with a reproducibility packet:

host rustc and cargo identity:
target crate edition and MSRV:
proc-macro dependency tree and lockfile hash:
positive/downstream commands:
UI fixtures and required diagnostic phrases:
generated-code sample for generic input:
clean and incremental timing sample:
filesystem/network restrictions:
grammar and runtime compatibility owner:

That packet makes a compiler extension governable. A maintainer can update the parser, compiler, or edition and know which claims to retest. A security reviewer can identify code that executes during builds. A downstream team can distinguish a diagnostic regression from an intended language change.

The packet is evidence for maintenance, not a promise that every compiler presentation remains frozen.

Durable rules

• A proc macro is host-executed compiler extension code and a supply-chain execution surface.
• Preserve tokens and spans through parse, domain validation, generation, and diagnostics; never use strings as the syntax protocol.
• Reject unknown, duplicate, and unsupported input deliberately; preserve caller generics and intent you do not own.
• Attach failures to user-owned tokens and accumulate independent errors in deterministic order.
• Generate small, explicit, fully qualified Rust while acknowledging dependency-renaming and limited-hygiene trade-offs.
• Test domain logic, downstream behavior, compile failures, expansion shape, editions, MSRV, and build cost as distinct evidence.
• Prefer stable APIs and spanned compile errors over an architecture dependent on unstable diagnostic facilities.
• Keep runtime behavior in ordinary libraries; make the proc-macro crate a thin, auditable adapter.

Sources and version notes

Verified on 2026-07-12 against the stable Rust Reference and standard-library proc-macro model, plus Rust 1.97.0 and the Rust 1.85.0 MSRV using proc-macro2 1.0.106, quote 1.0.45, and syn 2.0.117. Exact diagnostics and third-party parser behavior are version-sensitive.

• The Rust Reference: procedural macros
• Standard library proc_macro documentation
• Standard library TokenStream documentation
• Standard library Span documentation
• Cargo reference: build dependencies and compile-time cost considerations

With a trustworthy pipeline in place, the next design question is no longer how tokens become an implementation. It is which public macro form—derive, attribute, or function-like—should own a particular language, and how that language can evolve without surprising callers.