Slices, Strings, Bytes, and UTF-8

Seven bytes, three scalar values, one user-perceived cluster question

Consider this valid Rust string:

Aé🦀

Its UTF-8 bytes are:

41  C3 A9  F0 9F A6 80

The value has seven bytes and three Unicode scalar values. Byte offsets 0, 1, 3, and 7 are string boundaries; offset 2 lands inside the two-byte encoding of é. Rust therefore permits text.get(..3) and returns None from text.get(..2). It does not let text[2] pretend that “index 2” has one universal meaning.

That refusal is the foundation of reliable data APIs. A byte offset is not a scalar-value ordinal. A Unicode scalar value is not necessarily a grapheme cluster that a user perceives as one character. An operating-system string is not necessarily Unicode. A C string has a terminator and forbids interior NUL but does not thereby acquire UTF-8 encoding. Each boundary must retain the representation that the producer actually supplied until validation justifies a stronger type.

Data-view rule: accept and preserve the weakest accurate representation at a boundary, validate the invariants required by the next operation, and allocate or perform lossy conversion only as an explicit policy decision.

Arrays own a fixed count; slices borrow a sequence

[T; N] is a sized array containing exactly N elements inline. Its length participates in its type, so [u8; 16] and [u8; 32] are distinct types. This is valuable for fixed protocol fields, hashes, keys, SIMD-width blocks, and APIs where an exact count is an invariant rather than runtime metadata.

[T] is a dynamically sized slice type. Code normally uses it behind a pointer-like form:

• &[T] shares a borrowed sequence;
• &mut [T] exclusively borrows a mutable sequence;
• Box<[T]> owns a fixed-length heap sequence;
• Arc<[T]> shares ownership of a fixed-length sequence.

A slice carries enough metadata to locate a contiguous run of initialized elements. On current implementations a slice reference is commonly represented by a data pointer and length, but the Reference only guarantees that pointers to dynamically sized types are at least pointer-sized; do not freeze an observed two-word form into an ABI without an explicit contract. Semantically, the length is in elements, not bytes. For &[u32], len() == 4 describes four integers.

Slicing creates a view; it does not copy elements:

fn checksum_body(packet: &[u8]) -> u32 {
    packet.get(4..).unwrap_or_default().iter().map(|&b| u32::from(b)).sum()
}

The returned or local view cannot outlive the backing owner. Bounds checks protect safe indexing and slicing. Optimizers can often remove redundant checks in well-structured loops, but safety should not be weakened in anticipation of that result. If bounds checks appear in a profile, reshape the loop or validate once around a safe slice before considering unsafe access.

Mutable slicing additionally proves exclusivity. split_at_mut can derive two disjoint mutable slices because its implementation establishes non-overlap. Manually creating overlapping &mut slices would violate aliasing even if different code paths “usually” touch different elements. Express partitions through safe library operations or isolate a rigorously tested unsafe primitive.

str is a validated UTF-8 slice

The unsized str type has the same layout as [u8] and adds a permanent validity invariant: its bytes are valid UTF-8. &str is a borrowed text view; String owns a growable UTF-8 buffer. Both may contain \0; neither is NUL-terminated by definition.

Once bytes have become &str, downstream code may rely on UTF-8 validity without rechecking. That is the value of boundary validation:

let name = std::str::from_utf8(name_bytes)?;

The conversion scans only as required to validate and returns a borrowed view into the original bytes. It does not transcode or allocate. String::from_utf8 performs the analogous check while taking ownership of a Vec<u8>; on success it can reuse that allocation. Choose based on who should own the buffer after validation.

Unsafe constructors such as from_utf8_unchecked require the caller to prove all bytes are valid UTF-8 for the entire lifetime of the resulting string view. Violating this invariant is undefined behavior, not merely garbled output. If bytes originate from a network, file, device, C API, or unchecked mutation, the validation cost is normally part of accepting the boundary. Remove it only with upstream evidence strong enough to maintain the invariant structurally.

String exposes capacity because growth may allocate. push_str appends valid UTF-8 without needing to revalidate the existing string. Reserving can reduce reallocations when an upper bound is trustworthy; reserving attacker-controlled lengths without a limit can turn a length field into memory exhaustion. Treat capacity planning and input limits as the same production design.

Byte offsets, scalar values, and grapheme clusters answer different questions

Rust’s char is a Unicode scalar value, not a byte and not a user-perceived character. text.chars() iterates scalar values. text.char_indices() pairs each scalar with its starting byte offset, which is useful when a later operation must slice the original str.

The fixture records:

let text = "Aé🦀";
assert_eq!(text.as_bytes(), [0x41, 0xc3, 0xa9, 0xf0, 0x9f, 0xa6, 0x80]);
assert_eq!(text.char_indices().collect::<Vec<_>>(), [(0, 'A'), (1, 'é'), (3, '🦀')]);
assert!(text.get(..2).is_none());
assert_eq!(text.get(..3), Some("Aé"));

Several lengths are legitimate:

• text.len() is seven UTF-8 bytes;
• text.chars().count() is three scalar values and requires a scan;
• displayed width depends on fonts, terminal rules, combining behavior, and context;
• grapheme-cluster count requires Unicode segmentation policy and is not provided as a primitive str operation.

For example, a letter plus a combining mark may be two scalar values but one extended grapheme cluster. Emoji sequences can contain several scalar values joined into one displayed unit. “Maximum 20 characters” is incomplete until the product defines whether it limits bytes for storage, scalar values for a protocol, grapheme clusters for user editing, or display columns for a terminal.

Do not use Unicode normalization, case conversion, or segmentation casually in security-sensitive identifiers. Visually similar strings can have distinct code points; lowercase conversion can change length; normalization policy affects equality and signatures. Define canonicalization at one boundary, use a maintained Unicode implementation when needed, version the policy for persistent identities, and retain the original input when audit or display requirements demand it.

The boundary map now has a precise reading: the data-plus-length views borrow storage, the marked offsets are UTF-8 boundaries rather than character indexes, and the four input lanes retain different validity contracts.

Index with the unit the operation needs

Rust deliberately omits integer indexing that returns one char from str. UTF-8 scalar values occupy variable numbers of bytes, so such indexing could not be both constant-time and semantically obvious. Use an operation that names its unit:

• as_bytes().get(i) for one byte;
• get(range) for an optional substring on valid byte boundaries;
• is_char_boundary(i) before a byte-range split;
• char_indices() when scalar values and source offsets are both needed;
• chars().nth(n) for the nth scalar value, accepting linear traversal;
• a Unicode segmentation library and explicit version for grapheme operations;
• a domain parser for tokens, fields, lines, paths, or protocol units.

Repeated chars().nth(i) in a loop can make a linear scan quadratic. Iterate once, collect only if random access is genuinely required, or build an index whose storage and invalidation cost are justified. Editing systems often use ropes, piece tables, or byte buffers with boundary indexes because String alone does not make arbitrary middle edits cheap.

Panicking range indexing such as &text[a..b] is appropriate when a and b are already proven protocol or parser invariants. At external boundaries, get makes invalid offsets an ordinary error. Do not silence the distinction by clamping to nearby boundaries unless the product explicitly wants truncation; silently moving a security token or log cursor can be worse than rejection.

Parse bytes first and return views when lifetime permits

The memory-views-lab frame format is:

[kind: u8][name_len: u16 big-endian][name UTF-8][opaque payload bytes]

Its parser validates the structural boundary before the text boundary:

pub fn parse_frame(input: &[u8]) -> Result<Frame<'_>, ParseError> {
    let (&kind, rest) = input.split_first().ok_or(ParseError::TruncatedHeader)?;
    let (&name_hi, rest) = rest.split_first().ok_or(ParseError::TruncatedHeader)?;
    let (&name_lo, body) = rest.split_first().ok_or(ParseError::TruncatedHeader)?;
    let name_len = usize::from(u16::from_be_bytes([name_hi, name_lo]));

    let name_bytes = body.get(..name_len).ok_or(ParseError::NamePastEnd {
        declared: name_len,
        available: body.len(),
    })?;
    let payload = &body[name_len..];
    let name = std::str::from_utf8(name_bytes).map_err(|error| {
        ParseError::InvalidNameUtf8 { valid_up_to: error.valid_up_to() }
    })?;

    Ok(Frame { kind, name, payload })
}

The result borrows both fields from input. name is now proven UTF-8; payload remains arbitrary bytes, including 0xff and NUL. No allocation or transcoding occurs. The lifetime prevents retaining the frame after the input owner disappears.

Validation order matters. The parser obtains the three-byte header without unchecked indexing, converts the declared length to a host size, verifies that the range exists, and only then asks UTF-8 validation to inspect it. A more complex protocol must also check arithmetic overflow, maximum lengths, aggregate limits, recursion depth, duplicate fields, canonical encodings, and trailing-data policy before allocation or expensive work.

Zero-copy is not automatically faster. Borrowed frames keep the entire input buffer alive, complicate buffering and asynchronous handoff, and can pin a large network slab for one tiny field. Owning selected fields costs copies but may shorten retention and simplify queues. A hybrid parser can borrow during immediate routing, then copy only the values that cross an ownership boundary. Measure retained memory and end-to-end throughput, not just allocation count in the parsing function.

Streaming changes the lifetime problem

A borrowed field is valid only while its bytes remain stable. A streaming decoder that compacts or refills its buffer cannot hand out long-lived &str views into regions it may overwrite. Credible designs include:

• process each borrowed frame synchronously before refill;
• freeze reference-counted byte chunks and return range-backed views;
• copy selected fields into owned domain values;
• have the parser call a consumer while the borrow is active;
• use offsets into an owner that remains explicit in the result.

Each design pays somewhere: copies, allocation, reference counts, API complexity, or constrained control flow. Unsafe self-referential structures are rarely the first answer. Make the backing owner visible and ensure cancellation or queueing cannot retain unbounded buffers.

OsStr preserves platform strings

File names, command-line arguments, environment values, and other operating-system strings are not universally valid UTF-8. Rust uses OsStr for borrowed platform strings and OsString for owned ones. Their internal encoding is platform-specific and intentionally not a portable wire or storage format.

Accept &Path or &OsStr when the operation is about OS identity, not human text:

fn open_snapshot(path: &std::path::Path) -> std::io::Result<File> {
    File::open(path)
}

Changing this API to &str rejects valid platform paths on systems where path names need not be Unicode. Converting with to_string_lossy may merge distinct inputs through replacement characters, so it must not drive authorization, cache keys, file selection, or round trips. Lossy display can be appropriate for diagnostics if the UI clearly treats it as a rendering and structured logs retain a safe lossless identifier or platform representation under policy.

On Unix, platform extension traits expose path/OS strings as bytes. On Windows, extension traits convert to and from wide code units. Cross-platform code should stay with Path/OsStr for as long as possible and isolate native conversion in platform adapters. OsStr::as_encoded_bytes exposes an unspecified self-synchronizing encoding with strict reconstruction scope; those bytes should not be sent over a network or persisted as a cross-version format.

Display also has a security dimension. Control characters, bidirectional text, terminal escape sequences, and visually confusable names can deceive logs and operators. Escape or structure untrusted names for their destination, retain the original value for operations, and avoid treating a pretty rendering as a canonical identifier.

CStr preserves the C string contract, not its encoding

A C string is a sequence of non-NUL bytes followed by a NUL terminator. CStr is the borrowed Rust view of that representation; CString is an owned form whose constructor rejects interior NUL. Neither type asserts UTF-8. Converting a received CStr with to_str validates UTF-8 and can fail; to_string_lossy substitutes invalid sequences and may allocate.

The fixture demonstrates an outbound check:

pub fn to_c_string(text: &str) -> Result<CString, NulError> {
    CString::new(text)
}

assert!(to_c_string("worker-7").is_ok());
assert!(to_c_string("worker\0admin").is_err());

Rust strings may contain NUL, so conversion is fallible even when the source is valid UTF-8. The FFI contract must additionally name the C-side encoding, pointer nullability, maximum readable extent, ownership, mutation, retention, and deallocator. *const c_char alone answers none of these.

For inbound pointers, create a CStr only when the pointer is valid and a terminator is guaranteed within an accessible bound. Unbounded terminator search on an untrusted or invalid pointer can read outside the allocation. Prefer APIs that carry explicit lengths, or validate within a documented maximum before creating safe views. Copy when the foreign owner may mutate or free the bytes after the call.

Do not pass String::as_ptr() as a C string. It is not guaranteed NUL-terminated, and its UTF-8 buffer may contain interior NUL. Conversely, do not assume a C string is UTF-8 merely because its bytes display correctly in one locale.

Lossy conversion is a product decision

Lossy conversion replaces data that cannot be represented, commonly with the Unicode replacement character. It is useful for best-effort diagnostics, UI labels, and telemetry where continued display is more important than round-trip identity. It is dangerous for:

• authentication and authorization subjects;
• file and process selection;
• cache or database keys;
• signatures, hashes, and deduplication;
• protocol routing;
• values that must be edited and written back unchanged.

The replacement can make two distinct byte sequences produce the same visible string. A log should therefore distinguish a lossy rendering from a stable escaped byte representation or opaque identifier. Metrics labels need cardinality controls and must not ingest arbitrary full inputs merely to avoid loss.

Expose loss in the API. A function returning Cow<'_, str> from a lossy conversion may borrow when valid and allocate when replacement is needed; callers should not assume allocation-free behavior. Names such as display_lossy or an explicit result structure are clearer than a generic to_text.

Audit four boundaries without collapsing them

Suppose one agent-launch API accepts:

1. a user-supplied display name;
2. an executable path from the OS;
3. a wire frame containing a UTF-8 route and binary payload;
4. a C library label.

One String parameter for all four erases necessary contracts. A stronger design uses distinct views:

struct LaunchRequest<'a> {
    display_name: &'a str,
    executable: &'a std::path::Path,
    route: &'a str,
    payload: &'a [u8],
    native_label: &'a std::ffi::CStr,
}

This is only the validated inner request. Boundary adapters still have different work.

User text

Accept UTF-8 through the application protocol, define byte and user-facing length limits separately, decide normalization and prohibited-control policy, and retain the original if audit fidelity matters. Grapheme-aware truncation belongs in the presentation/product layer, not an arbitrary byte slice.

OS path

Receive PathBuf or OsString from the environment, preserve it losslessly for lookup, and escape it for diagnostics. Do not require Unicode merely to make JSON logging convenient; provide a structured platform representation or a deliberately lossy display field.

Wire data

Start with &[u8], check message and field limits, validate only fields declared UTF-8, and leave opaque payload bytes opaque. Borrow views during immediate processing; copy or retain a bounded backing owner for queued work. Reject invalid canonical forms when signatures or equality depend on them.

C string

Use &CStr only after pointer and terminator safety is established, then apply the encoding named by the foreign API. Copy before the foreign owner can invalidate storage. Use CString outbound and handle interior NUL explicitly.

The types now document which operations are legal. route can use string search; payload cannot. executable remains a platform path; native_label remains terminator-aware. Allocation appears only when ownership crosses a lifetime or conversion policy requires it.

Production costs and failure behavior

Allocation and retention

Repeated to_string, to_owned, format!, and lossy conversion can dominate parser and logging paths. Instrument allocations where the workload matters. Also measure buffer retention: a 20-byte borrowed field can keep a multi-megabyte chunk alive. Pooling reduces allocation frequency but may preserve high-water capacity and secrets; bound pools and scrub data according to the threat model.

Denial of service

Validate lengths before reservation, multiplication, normalization, regex work, or decompression. Apply per-field, per-message, nesting, and aggregate budgets. UTF-8 validation is linear, but repeated rescans can multiply cost. Validate once, keep &str or an owned string as the proof, and avoid converting back to bytes and revalidating without cause.

Observability

Classify errors: truncated structure, out-of-range length, invalid UTF-8 with valid_up_to, interior NUL, unsupported encoding, and policy rejection are different incidents. Report offsets and bounded escaped samples, not entire secret-bearing payloads. Counters should use stable low-cardinality reasons.

Compatibility

Protocol fields must state byte encoding and normalization, not “string.” Platform paths should not be serialized using OsStr’s internal encoded bytes. C APIs should document locale or explicit encoding. Persistent indexes that change Unicode normalization or case-fold versions need a migration strategy.

Panic behavior

Range indexing panics on out-of-bounds or non-boundary offsets. That can be appropriate behind a proven invariant but should not turn malformed external data into process failure. Boundary parsers should return typed errors. Test invalid lengths, incomplete UTF-8, maximal scalars, NUL, empty values, combining sequences, and platform-native non-Unicode cases.

Alternatives and their costs

No row has a universal winner. The starting type preserves information; the alternative buys lifetime, mutability, compatibility, or access characteristics at a measurable cost.

Failure modes to reject in review

• Accepting String for arbitrary wire bytes and forcing invalid data through lossy conversion.
• Accepting &str for OS paths and rejecting valid platform names.
• Treating len() on str as a character or display-width count.
• Slicing text at an unchecked byte offset from user input.
• Repeatedly calling chars().nth to simulate random access.
• Assuming every char corresponds to one user-perceived grapheme.
• Persisting OsStr::as_encoded_bytes() as a portable format.
• Treating CStr as UTF-8 without validation and an encoding contract.
• Passing String::as_ptr() to an API expecting a terminated C string.
• Constructing CStr from an untrusted pointer without a bounded validity proof.
• Calling conversion “zero-copy” while a tiny view retains an oversized input slab.
• Reserving attacker-declared lengths before applying limits.
• Using lossy text as an identity, authorization subject, key, signature input, or round-trip value.
• Publishing an observed fat-pointer width as a stable ABI guarantee.

Exercise: audit a four-string API

Review this proposed interface:

fn launch(name: String, executable: String, packet: String, c_label: String) -> Result<(), Error>;

Produce a replacement API and boundary plan that:

1. Defines the units and encoding of every input.
2. Uses &str or String only for values whose UTF-8 invariant is established.
3. Preserves the executable as Path/OsStr without lossy identity conversion.
4. Parses the packet from bytes, validates bounds before allocation, and leaves its binary payload as bytes.
5. Uses CStr/CString at the C boundary and handles interior NUL and non-UTF-8 encoding.
6. States which returned views borrow which owner and when copying becomes necessary.
7. Defines byte, scalar, grapheme, and display limits only where each is actually required.
8. Specifies normalization, truncation, logging, escaping, and lossy-display policy.
9. Tests malformed lengths, invalid UTF-8, combining sequences, embedded NUL, native non-Unicode paths, empty data, and maximum allowed sizes.
10. Compares allocation count, buffer retention, error observability, and compatibility with the original design.

Approval requires more than changing parameter types. The implementation must preserve each representation through the correct boundary adapter and demonstrate that malformed input fails before unsafe access or unbounded work.

Boundary review card

• Is the value bytes, UTF-8 text, platform text, a path, or a C string?
• Does the API need ownership, or will a borrowed view cover the operation?
• What owner keeps every returned view alive?
• Is length measured in bytes, elements, scalar values, grapheme clusters, or display columns?
• Are substring offsets proven UTF-8 boundaries?
• Is validation performed once before construction of the stronger type?
• Are declared lengths bounded before allocation and expensive work?
• Can a small borrow retain a disproportionately large buffer?
• Does streaming refill invalidate outstanding views?
• Is lossy conversion restricted to a named display or diagnostic path?
• Are OS values preserved as OsStr/Path across operational decisions?
• Does a C boundary specify terminator, length, encoding, ownership, and lifetime?
• Are native or internal encodings kept out of portable storage and wire formats?
• Do logs escape untrusted text and avoid leaking payloads?

What engineers may rely on

Arrays own a compile-time element count; slices are dynamically sized sequences usually accessed through a pointer-like owner or borrow. str has the layout of [u8] plus a permanent UTF-8 validity invariant. String owns growable UTF-8 storage. Safe string slicing requires byte offsets that are also UTF-8 boundaries, while scalar values and grapheme clusters remain different units.

OsStr preserves platform strings but does not define a portable byte encoding. CStr preserves NUL-terminated C representation but does not define text encoding. Parsing from bytes should establish structural bounds before textual validity and may return borrowed views when the backing owner and processing lifetime align. Allocation, copying, and lossy conversion are sometimes correct; they should be visible consequences of lifetime and product policy rather than hidden conveniences.

These decisions set up Chapter 29. Collections do not merely store values: capacity changes addresses, key borrowing affects lookup APIs, and iteration order can become a compatibility concern. The view and ownership contracts established here determine which collection operations are safe and which costs are observable.

Sources and version notes

• Rust Reference: array and slice types and slice types
• Rust Reference: type layout for slices, str, and pointers
• Standard library: str, String, and from_utf8
• Standard library: OsStr and OsString
• Standard library: CStr and CString
• Unicode Standard Annex #29: Unicode Text Segmentation
• The memory-views-lab uses Rust 2024 Edition, was written for pinned Rust 1.97.0, and declares Rust 1.85 as its MSRV. Its parser is dependency-free and safe-code-only; production Unicode segmentation, normalization, and platform adapters require separately versioned policies and tests.