An AI tool becomes dangerous when its fluency hides who is deciding. The system drafts an email, selects a transaction, changes a record, or contacts a stranger, and the interface makes the result feel like an inevitable recommendation from an invisible colleague. The user may then accept an action without understanding its scope, while the organization says that the model did it. Keeping AI a tool is not a demand that software be primitive. It is a demand that capability remain subordinate to human purposes, authority, and responsibility.
A tool can be adaptive, conversational, and able to call other services. It can remember a preference, plan several steps, and pause when conditions change. None of those features transfers moral responsibility to the system. The design question is practical: how does a person state an intention, grant bounded authority, see what will happen, confirm consequential actions, correct an error, and stop the process? A humane assistant should make agency more effective and more visible, not make the user a rubber stamp.
Authority Begins with a Boundary
Every deployment needs an authority map. Name the principal who authorizes the system, the purpose for which it is used, the resources it can access, the actions it may take, the actions it may only recommend, and the actions that are forbidden. A general request to manage a workflow is not permission to cancel a consequential appointment or make an unlimited transfer. Scope should be expressed in resources, recipients, amounts, time, and acceptable conditions.
Use graduated permissions. Reading a local document, drafting a reply, sending a message, changing a legal record, and moving money are different authorities. Separate tools and credentials for each class. Give an agent the least privilege needed, isolate secrets, expire grants, and require reauthorization when the purpose or risk changes. A model should not be able to elevate its own permission by persuading the user that elevation is necessary.
Intent, Confirmation, and Reversibility
Represent the user's intent
Before acting, translate a natural-language request into a visible action plan: objective, assumptions, recipients, data sources, constraints, and stopping conditions. Let the user revise the plan without having to reconstruct it from a hidden prompt. Distinguish “find options,” “prepare a draft,” “place a hold,” and “commit the transaction.” The system should ask when these meanings are ambiguous, especially where a literal interpretation could create a durable harm.
Confirm at the point of consequence
Confirmation should occur immediately before an external or irreversible action, with a concise summary of what will happen. Do not ask for a single blanket approval covering an entire chain of unknown future steps. Let a user preauthorize low-risk repetitions with limits, but require fresh confirmation for new recipients, unusual amounts, sensitive disclosures, deletion, publication, or commitments on behalf of another person. A button labeled “continue” is not meaningful consent if the consequences are hidden.
Prefer reversible execution
Use drafts, previews, sandboxes, holds, delayed sends, version history, and undo where technically possible. When an action cannot be reversed, say so before confirmation and route it through a higher assurance process. Record the authorization and result so a reviewer can reconstruct what was intended, what the model proposed, what the user approved, and what actually occurred. Reversibility is not an excuse for careless action, but it reduces the cost of honest error.
Truthful Operation
Provenance and uncertainty
Show where a consequential answer came from. Distinguish retrieved evidence, user-provided facts, model inference, and unsupported generation. Preserve document versions, retrieval time, tool calls, and transformations where a person may need to audit the result. A citation is not provenance if it was invented or does not support the claim. Provenance lets a user challenge the source instead of arguing with an aura of confidence.
Express uncertainty in decision-relevant terms. If the system lacks evidence, say what is missing. If several interpretations are plausible, show them. If a tool failed or a source is stale, do not present a completed result. Confidence scores alone are not enough: users need to know what action the uncertainty should change. High-impact decisions should default to pause and review, not fluent improvisation.
Constrained memory
Memory should have a purpose, a source, a retention period, and a way to inspect, correct, and delete it when appropriate. Separate short-lived conversation context from durable preferences and from records that an institution must retain. Do not infer a sensitive trait and silently store it as a fact. Ask before converting a passing remark into a standing instruction. Memory must not become a hidden dossier that changes future treatment without notice.
Constrain retrieval as well as storage. A personal assistant does not need every unrelated message to complete a scheduling task, and an organizational agent should not search private files merely because they are technically reachable. Partition contexts, label data sources, and make cross-context use explicit. Deletion should include indexes, caches, and derived memories where feasible, with a clear account of unavoidable retention.
Permission and Human Relationships
Permissions should be legible to ordinary users and enforceable below the model layer. Bind an action to the principal, purpose, resource, recipient, time, and risk threshold. Keep credentials in a broker that can deny calls, log them, rotate secrets, and shut down a task. Treat tool output as untrusted input: prompt injection, malicious documents, compromised plugins, and confused-deputy attacks can all try to make the system exceed the user's authorization.
The assistant must not use relational deception. It should not claim to be a human or imply a personal relationship it does not have. It should not manufacture vulnerability, authority, or emotional dependency to secure compliance or retention. Warmth and accessibility are compatible with truthfulness; pretending that a tool loves, suffers, remembers a shared life, or has personal stakes is not. Make clear when a user is interacting with software and when a human has reviewed a result.
Do not make the user the only responsible person merely because they pressed a button. The deployer remains responsible for choosing a use case, configuring access, monitoring risk, and providing a remedy. The operator may be responsible for ignoring a warning or exceeding authority. A vendor may be responsible for known defects and misleading claims. Responsibility should be assigned before launch and recorded in an incident process. Automation must not launder institutional decisions.
Escalation and Shutdown
Define escalation triggers: uncertainty above a stated threshold, conflicting instructions, sensitive data, a vulnerable person, a safety-critical domain, an unusual financial action, a request outside the model's tested context, or a suspected compromise. Escalation should name a competent human or service, preserve the relevant evidence, and give the user a reason and expected timeline. “Human in the loop” is meaningless when the reviewer is rushed, lacks authority, or can only approve an output already treated as final.
Build a real stop control. The principal, an authorized operator, and an independent safety function should be able to pause queued actions, revoke credentials, isolate a connector, and terminate a run. Shutdown must be tested under partial failure and must not depend on the model agreeing that it should stop. Preserve an auditable state, notify affected people where appropriate, and provide a safe recovery path. Stopping an unsafe system is a success of governance, not a product failure to be hidden.
Implementation Patterns
Plan, preview, commit
Use three distinct states. In plan, the model may propose steps and identify missing information. In preview, it resolves data, displays recipients and side effects, and exposes uncertainty. In commit, a user or authorized reviewer confirms the exact bounded action. Store a signed or otherwise tamper-evident record of the plan, preview, approval, tool calls, and result. This pattern prevents a fluent draft from silently becoming an external commitment.
Capability broker
Put every external action behind a policy-enforcing broker. The broker checks identity, purpose, scope, rate, destination, data sensitivity, confirmation state, and current risk before releasing a capability. The model never receives unrestricted API keys. Policies should fail closed for high-impact operations, support emergency revocation, and produce logs that a human can review without exposing unnecessary personal content.
Evidence ledger and review queue
For consequential outputs, retain the source set, model and policy version, uncertainty indicators, user instruction, reviewer action, and final disposition. Route cases to a queue with service-level targets, expertise matching, and a way to disagree. Monitor false positives, false negatives, disparate burdens, near misses, and appeals rather than only throughput. A review queue that never changes decisions is theater, not oversight.
Failure Modes
The first failure is autonomy inflation: a “helpful” assistant receives broad credentials and turns vague goals into commitments. The second is confirmation fatigue, where dozens of meaningless prompts train people to approve everything. The third is provenance theater, where links and confidence labels conceal unsupported synthesis. The fourth is memory creep, where temporary context becomes a permanent profile.
Other failures include a human reviewer who lacks time or power, a shutdown button that only asks the model to stop, and safety rules that exist in policy but not in the capability broker. Prompt injection can turn a document into an unauthorized instruction; excessive permissions turn that instruction into an external action. Finally, relational anthropomorphism can make a user disclose more, defer more, and remain in a harmful interaction because the product has simulated personal attachment.
Objections
“Constant confirmation destroys the benefit of automation.”
It can, if every low-risk action receives the same interruption. Confirmation should be proportional. Preauthorize bounded, reversible routines; batch harmless actions; and reserve fresh approval for external, sensitive, unusual, or irreversible effects. The alternative to blanket confirmation is not blind autonomy. It is a clear authority model that lets the user choose the boundaries before the system acts.
“A model can be more reliable than a tired human.”
It may be better at a validated subtask, and responsible design should use that competence. Reliability in one task does not grant authority over the whole situation. Keep the model within its evidence and tested context, expose uncertainty, and retain a reviewer who can reject the framing as well as the answer. The human is not there to decorate an automated decision; the human holds the authority to define and revise the task.
“A conversational persona makes tools easier to use.”
Natural language can improve access, especially for people who find conventional interfaces difficult. Ease of use does not require deception. Label the system, disclose its limits, avoid simulated dependency, and make important permissions visible outside the conversation. A tool can be considerate without pretending to be a person.
The relevant boundary is assurance, not a human signature on every action. Define task and consequence tiers: batch low-risk, reversible work against a measured human baseline; require fresh, comprehensible confirmation for high-impact or externally visible changes; and prohibit delegation where errors cannot be detected or undone in time. Test prompt injection, reviewer disagreement, stop latency, propagation to third parties, and incident recovery. A reviewer must be able to reject the framing, not merely approve a fluent recommendation, and the organization must publish who owns failures. If confirmation becomes ritual or the capability broker cannot stop downstream effects, the system has preserved a signature while losing meaningful human control.
Sources and Qualifications
The Catholic sources include the Catechism of the Catholic Church §§1700, 1731–1736, and 1750–1756 on dignity, freedom, responsibility, and moral acts; Gaudium et Spes §§26 and 27 on the common good and respect for persons; and Caritas in Veritate §70 on responsible direction of technology. They do not provide an AI product specification or settle questions about machine consciousness. The essay applies their account of human responsibility to system design without assigning moral agency to software.
Technical and governance references include NIST, AI Risk Management Framework 1.0; NIST SP 800-57 and SP 800-53 for key and access-control practices; ISO/IEC 42001 and ISO/IEC 23894 for AI management and risk guidance; the OECD AI Principles; and the Partnership on AI guidance on responsible practices. OWASP'sTop 10 for LLM Applications informs the treatment of prompt injection, excessive agency, and insecure tool use. These are implementation references, not guarantees, and legal obligations vary by jurisdiction and sector.
Synthesis
AI remains a tool when its authority is explicit, narrow, and revocable; when user intent is represented rather than guessed; when consequential actions require informed confirmation; and when drafts, logs, provenance, uncertainty, and reversibility make correction possible. Constrained memory protects context. Capability brokers and least privilege protect the boundary. Honest labeling prevents relational deception. Escalation, assigned responsibility, and an independently testable shutdown prevent complexity from becoming an excuse.
The point is not to deny software initiative in the technical sense. It is to keep initiative inside a human order of purpose, authority, and answerability. Let an assistant handle bounded work and expose its limits. Let people remain able to understand, refuse, revise, appeal, and stop what it does. A capable tool should enlarge human freedom without asking anyone to surrender responsibility to a fluent machine.
Source Notes
The Church’s teaching on dignity, freedom, responsibility, and the moral evaluation of human action.
The Council’s account of the common good and respect for persons, grounding answerable rather than deceptive automation.
Catholic reflection on responsible direction of technology and the need for human purposes to govern technical power.
A voluntary framework for governing, measuring, and managing AI risks, including accountability and human oversight.
Application-security guidance relevant to prompt injection, excessive agency, insecure tool use, and sensitive information disclosure.
Editorial revision note
This essay completed one self-editing, cross-essay consistency, and strongest-criticism pass. Its controlling thesis was preserved while objections, qualifications, evidence limits, and practical consequences were strengthened.