Senior Engineering Interview Handbook / Chapter 109
Build, Buy, or Partner
A production-engineering foundation chapter that follows one notification-platform decision through strategic differentiation, delivery pressure, operating burden, security and compliance review, vendor failure, partnership ownership, switching costs, and exit.
Page tools
The vendor can send messages. Who owns notification?
A product team has six weeks to add email, SMS, and push notifications for account activity, approvals, reminders, and security events. Established providers already operate the channel infrastructure. Building carrier connections, sender-reputation systems, and mobile delivery networks would consume the schedule before the first useful message reached a customer.
“Buy” appears to settle the question. Then the team tries to draw the system.
Who decides whether an event should become a message? Where do consent, preferences, templates, and localization live? Can a security alert wait behind a marketing reminder? If a provider accepts a request and the response times out, is it safe to send again? What can support tell a customer who received three messages or none? Which data crosses a regional boundary? How would the product leave the provider without losing delivery history or freezing every team that emits notifications?
The provider answer was easy because it named an implementation. The questions are harder because they reveal ownership.
Begin with the outcome, then divide the capability
Customers need messages that are timely, accurate, understandable, and consistent with their choices. The product also needs evidence of what it tried to send and a way to recover when delivery goes wrong. That outcome contains several different capabilities.
Channel machinery includes carrier rules, bounce handling, sender reputation, device gateways, and regional delivery connections. A specialist provider can usually operate this layer better than a product team with six weeks and no delivery expertise.
Product policy is different. The product knows which events deserve a message, which template and locale apply, whether the user has consented, how urgent the message is, and whether another channel is an acceptable substitute. Delegating those decisions can surrender part of the product itself.
Operating truth belongs somewhere too: message identity, current state, attempts, provider responses, audit history, cost, and the explanation visible to support. If the only record lives in a vendor console, every incident and customer question begins across an organizational boundary.
Finally, the product needs a change boundary. Providers fail, prices move, interfaces are deprecated, requirements change, and contracts end. A design that works only while the first agreement remains favorable has postponed part of the architecture.
For this system, the likely answer is mixed: buy channel delivery and build the small control plane that expresses product policy, records delivery state, and contains provider differences. A partner may supply a regional channel or a regulated distribution relationship that neither side can offer alone.
Put the boundary under pressure
“Core versus non-core” is too blunt for the decision. Authentication may not differentiate a product, yet losing control of account recovery can destroy trust. Search infrastructure may be commodity while ranking and relevance are the marketplace. Notification delivery may be external while consent and security-message policy remain firmly inside the product.
Five pressures make the argument inspectable.
Differentiation asks where the company learns something competitors cannot easily copy. For notifications, carrier connectivity is unlikely to be that place. Approval workflows, message timing, and the way a security event reaches a customer might be.
Delivery pressure asks what speed buys. A vendor can help the team learn whether customers value the feature before it commits to a large internal platform. Speed is evidence only when the integration remains good enough to teach the right lesson.
Operating burden continues long after launch. An internal system needs staffing, upgrades, capacity planning, on-call ownership, abuse controls, documentation, and support. A vendor integration still needs monitoring, incident response, contract and deprecation tracking, and engineers who understand its failure semantics.
Trust risk asks what data and authority cross the boundary. The team should map message contents, identifiers, retention, access, deletion, regional movement, and audit evidence, then involve qualified security, privacy, legal, or compliance owners where the domain requires them. Buying cannot substitute for that judgment.
Exit cost asks what must be preserved if the relationship ends: templates, preferences, suppression lists, delivery history, audit evidence, interfaces, and customer-visible behavior. The answer influences the initial data model, contract, and integration shape.
These pressures do not produce a score. A high-trust commodity may still need a carefully owned policy layer. A differentiating capability may begin with a vendor while the team learns. The purpose is to expose the assumption on which the recommendation depends.
Follow one message across the boundary
Suppose an account service emits a security event. The notification control plane assigns a stable message identifier, checks product policy and user preferences, selects an approved template and channel, and records an intent before calling the provider through a narrow adapter. Provider callbacks are translated into product states that support and monitoring can understand.
The state model needs room for uncertainty. queued, submitted, accepted,
delivered, and failed are useful only if their meanings are explicit; a
timeout after submission may leave the result unknown. Retrying blindly can
create duplicate security alerts. Refusing to retry can lose one. Stable
identifiers, an idempotent internal command boundary, provider deduplication
where it is actually guaranteed, and a policy for ambiguous outcomes are part
of the product design—not features purchased by implication.
The same boundary makes operations visible. The team can observe queue age, attempt count, provider latency and errors, unknown outcomes, opt-out correctness, cost per delivered message, and failures by workflow. Support can see that a password-change alert was accepted but not confirmed delivered without opening a provider console or reading raw logs.
This architecture is not free. The adapter can become a speculative abstraction stuffed with features no second provider shares. A second provider can double integration and operational work without improving recovery. The control plane should therefore own semantics the product genuinely needs and isolate only the differences likely to matter.
Make the first recommendation survive failure
Now the primary provider begins returning errors in one region. The right response depends on message class. A weekly reminder can wait while the queue is bounded and visible. A security event may justify a pretested alternate channel or provider. An approval request might need an in-product fallback. There is no honest global rule called “fail over notifications.”
Even a second provider does not erase uncertainty. If the first request timed out after acceptance, sending through the second may duplicate the message. Failover policy must consider urgency, duplicate harm, provider status, rate limits, data residency, and whether the product can tell the customer what happened. A contractual uptime promise says little about these choices.
The incident tests the sourcing decision. The provider owns restoration of its service. The product team owns detection, customer impact, queue behavior, degraded operation, support visibility, escalation, and communication. The boundary is sound when both sides can act without pretending the other owns the whole outcome.
Cost pressure tests it again. A price increase may be tolerable if the provider still removes substantial operating work; the preceding cost analysis should compare integration, on-call, support, audit, migration, and opportunity cost alongside the invoice. If a contract renewal or product change reverses the economics, the team needs enough evidence to reconsider rather than defend the original choice as an identity.
A partner creates one product and two operating organizations
A partnership is appropriate when the outcome depends on something the team cannot simply purchase: regional distribution, regulated access, licensed data, hardware reach, domain expertise, or an established customer relationship. In the notification system, a regional channel partner might provide access and local operating knowledge while the product supplies policy and customer context.
Shared value does not imply shared clarity. The agreement must name who owns data freshness and correction, interface changes, support intake, incident command, customer communication, security evidence, roadmap decisions, and the authority to disable a harmful integration. Otherwise every exception becomes a negotiation while the customer waits.
The engineering interface and the organizational interface should tell the same story. If one side can change message semantics without review, if neither side can observe the end-to-end path, or if support tickets can circulate without an accountable owner, the partnership has no operable boundary.
Leave a record that can survive renewal
The useful decision record is short enough to read during an incident and specific enough to challenge at renewal:
Capability and customer outcome:
Recommendation and options rejected:
Boundary:
- what we own: policy, data, user experience, state, observability, support
- what we delegate: implementation and operating responsibilities
- what requires shared decisions
Assumptions:
- differentiation and delivery pressure
- internal and external operating burden
- trust requirements and qualified reviewers
Failure:
- degraded behavior and ambiguous outcomes
- incident, escalation, and communication owners
Exit:
- data export and retention
- interface and migration path
- trigger that reopens the decision
Decision owner and review date:
For the modeled system, that record might recommend buying channel delivery, building notification policy and state, and using a partner only where access or distribution requires shared product work. It would reject a full internal delivery network because the team lacks both time and a strategic reason to own one. It would also reject direct calls from every product service because they scatter consent, retry, support, and exit behavior across the codebase.
The strongest part of the record is the reopening trigger. A provider sunset, unacceptable regional failure, a material price shift, an unmet audit need, or a new product capability may invalidate the old assumptions. Decisions become dangerous when their date disappears.
Choosing deliberately
Build when product advantage, unusual domain behavior, latency, privacy, or failure semantics require control that available providers and partners cannot offer. Count the permanent service, staffing, and opportunity costs, not only the initial implementation.
Buy when the problem is common, a provider is meaningfully stronger than the team can sustain, speed creates useful learning, and product policy, data, observability, support, and exit can remain under control. Treat the dependency as part of the architecture.
Partner when the result depends on another party’s access, distribution, data, hardware, expertise, or customer relationship. Make decision rights and the end-to-end operating model as explicit as the API.
Many good systems combine all three. The senior judgment lies in explaining why each boundary sits where it does and what evidence would move it.
Rehearse one decision until it changes shape
Choose a capability from a system you know: identity, payments, search, analytics, messaging, experimentation, or a data feed. Begin with the customer outcome and your first recommendation. Then force the recommendation through six changes:
- The provider is unavailable during the most important workflow.
- A sensitive data field must cross—or must stop crossing—the boundary.
- Usage grows enough to change the economics.
- Support must explain one disputed outcome to a customer.
- The vendor deprecates an interface with six months’ notice.
- A partner wants roadmap influence but will not own incident communication.
For each change, revise the boundary, degraded behavior, owner, and exit plan. End by stating the assumption that would make you choose differently.
In a system-design conversation, this exercise keeps the answer focused on interfaces, state, failure, and control rather than brand selection. In a project deep dive, it produces the rejected option and reopening trigger that make a decision credible. In a leadership conversation, it reveals how engineering, product, finance, security, legal, support, and an external party reached an operable agreement.
A build, buy, or partner recommendation is finished when the organization can operate the boundary on an ordinary day, defend it during failure, and change it when the assumptions move.
Related links
Continue reading
Full table of contents