Ethics of Superintelligence in Business: Decisions People Keep

Ethics of Superintelligence in Business: Decisions People Keep

The ethics of superintelligence in business comes down to one management question: which decisions stay with named people, with a role and a name attached, even when AI can recommend or execute better? The short answer is that people keep the decisions that affect other people's rights, commit the company in ways that are hard to reverse, set values and limits, and require accountability to someone. For each of those, the company needs a simple record of who decides, on what basis, and how the decision is reviewed.

Key takeaways

• Superintelligence is still a hypothesis under debate, but the habit of delegating grows with every gain in capability, so limits need to be set before that happens.
• Do not delegate decisions that affect people's rights, create commitments that are hard to reverse, open policy exceptions or define the limits of the AI itself.
• Responsibility does not pass to the model: it needs a named owner, a backup and a way for affected people to contest the outcome.
• Human approval only works as a control if the reviewer has evidence, time and the authority to say no.
• A responsibility register per decision, reviewed at agreed intervals, is more useful than a generic code of ethics.

What does the ethics of superintelligence have to do with your company today?

Superintelligence, in the sense used in the research literature, is a system that would outperform people at practically every relevant cognitive task. That is a hypothesis discussed by researchers, not a product a company can buy. What your operation uses today are capable but limited models and agents that make confident mistakes and depend on the context they receive. If you want to separate the concepts cleanly, the article on the difference between AGI, ASI and AI agents helps.

Why discuss the ethics of something that does not exist yet? Because the question that matters to a manager does not depend on the hypothesis coming true. The more capable the system, the stronger the temptation to let it decide, and the harder it becomes to notice when it should have stopped. Defining now what is not delegated is a cheap decision. Discovering it later, in the middle of an incident, is expensive.

It helps to separate three kinds of claims in this article. Official documentation, such as the NIST AI Risk Management Framework and the OECD AI Principles, addresses accountability, transparency and human oversight. Implementation experience is what you observe when agents are placed in real processes: the safest flows have explicit limits and records. Hypothesis is what is projected onto future systems, and it should be treated as such.

Which decisions should not be delegated to AI?

The list below is a starting point. Each company adjusts it to its own risk, but the logic is the same: the greater the impact on a person or on the company, and the harder it is to undo, the closer to a person the decision should stay.

• Decisions that affect a person's rights or opportunities: hiring, firing, denying credit, ending a contract, refusing a warranty claim. In Brazil, the LGPD gives people the right to request review of decisions made solely through automated processing of personal data. The article on automated decisions and Article 20 of the LGPD covers the point in detail.
• Policy exceptions: a refund outside the rule, a special discount, an extended deadline. Every exception sets a precedent, and precedent is a management decision.
• Significant legal and financial commitments: signing a contract, authorizing a payment, approving a budget above a limit the company defines.
• Crisis and bad-news communication: an accident, a data leak, a serious complaint, a notice to affected customers.
• Evaluating people: performance, promotion, warnings. AI can organize the information, but judgment about a person belongs to whoever answers for that person.
• Using personal data for a new purpose, especially sensitive data. Changing the purpose is a decision for the data controller, not for the workflow.
• Defining the limits of the AI itself: which data it can access, which tools it uses, which actions it can take on its own.
• Expanding, pausing or shutting down an AI system in production.
• Values and priorities: what the company will not do even when it would be profitable or efficient.

Note that the list does not stop AI from taking part. It can pull the history, summarize the case, compare it with the policy and suggest an option. What stays with the person is the final decision and the responsibility for it.

Why should people keep the decision even when the AI is right?

First, because the company, represented by people, is the one that answers to the customer, the regulator and the courts. A model is accountable to no one. Even the ANPD, Brazil's data protection authority, addresses data processing agents, not systems.

Second, because being right on average does not protect the individual case. A system can handle many conversations well and still fail on exactly the case that matters most, the one outside the pattern. The person deciding needs to be in a position to notice what does not fit.

Third, because human oversight degrades. When the system is right almost every time, the reviewer starts clicking approve without reading. At that point the company has a human signature without human control. This risk grows with AI capability, which is why process design matters more than the promise that someone will be watching.

Finally, there is institutional learning. Hard decisions teach a company about its own values. If everything is delegated, nobody can explain why the company decides the way it does.

How do you record responsibility for each decision?

A generic code of ethics does not solve this. What works is a short record per decision that anyone in the operation can read. Suggested fields:

• Decision: described in one sentence, in business language.
• Owner: the role and name of the person who decides, plus a backup for absences.
• Role of the AI: inform, recommend, prepare a draft, or execute within a written limit.
• Escalation trigger: what forces the AI to stop and call the owner, such as a request outside policy, a serious complaint, sensitive data or low confidence.
• Minimum evidence: what the owner must see before deciding, such as history, the applicable policy and the recommendation with its reasoning.
• Decision log: what the AI proposed, what the person decided, when, and whether the two differed.
• Contestation channel: how an affected person asks for review and who handles it.
• Review: how often the record is re-examined and what triggers an earlier review, such as an incident or a model change.

Two practical precautions. The first is least privilege: the agent only receives the permissions it needs for the role described, a principle covered in OWASP's authorization guidance. The second is keeping a log of disagreements. When the human disagrees with the AI often, or never does, both patterns deserve attention.

What does this look like in a real case? (illustrative scenario)

Imagine, as an illustration, a services company that uses a WhatsApp agent to handle cancellation and refund requests. The agent explains the policy, collects the case details, checks the history and prepares a summary. So far, everything is within the written limit.

A customer asks for a refund outside the policy deadline, citing a genuine problem. The agent promises nothing. It tells the customer that a person will review the case and sends the finance manager a summary with the history, the applicable rule and a recommendation with its reasoning. The manager decides to grant an exception, and the record stores the AI's recommendation, the decision, the date and the reason. Because the exception sets a precedent, it goes into the periodic review, where management assesses whether the rule should change.

The point of the scenario is not the decision itself but the path: the agent knew when to stop, the person had evidence, the decision was recorded and the customer knew someone was responsible. To see what is usually automated and what stays with the team in this kind of channel, read the article on AI customer support: what to automate and what to keep human.

What plan can you follow to put this in place without bureaucracy?

The sequence below follows the working method we use: map, limit, pilot and hand over. It fits a small company and scales to a larger one.

• Map: list the decisions made today in the process that will receive AI, who makes them and on what basis. Also record the ones nobody formally owns, because they are the riskiest.
• Classify: for each decision, ask whether it affects people's rights, whether it is hard to reverse and whether it sets a precedent. Yes answers point to a decision that stays with a person.
• Limit: write down the AI's role in each decision, the data it can access and the actions it can take. Anything not written down is prohibited by default.
• Name: define an owner and a backup for each reserved decision, with real authority to reject the AI's recommendation.
• Pilot: run one real flow, logging proposals, decisions and disagreements, with a success criterion agreed before it starts.
• Review: examine the logs at set intervals, adjust escalation triggers and check that human review is still attentive.
• Hand over: give the team the register, the limits and the review procedure, and decide with management whether the pilot expands, adjusts or stops.

In work with companies, this method has a practical consequence: the pilot starts small, with a single flow, because it is easier to validate the division of responsibilities in one process than in ten at once. When a managed operation is needed, it is run with Be Human, always with logs and human approval at the agreed points.

How do you know if human oversight has become a formality?

A few signs indicate that the decision has moved to the AI in practice, even if on paper it still sits with a person:

• The reviewer approves everything in quick succession and almost never changes or rejects the recommendation.
• Nobody can explain why the AI recommended something, and the evidence does not appear on the decision-maker's screen.
• The registered owner changed roles or left the company and nobody updated the record.
• The team says it is not worth overriding the system because it creates extra work.
• Exceptions start being decided by the workflow itself, without escalation, because the rule became ambiguous.
• The affected person does not know how to ask for review.

If two or more of these signs appear, go back to the register, reduce the AI's scope at that point and rebuild the minimum evidence. Also follow the Brazilian regulatory debate, which keeps moving: the article on Brazil AI regulation and what companies should watch points to what to monitor.

Delegating the task is an efficiency decision. Delegating the responsibility is not possible.

Frequently asked questions

Does superintelligence already exist, and do I need to worry now? No system recognized as superintelligent is in use in companies, and the topic remains a research hypothesis. Even so, defining decisions reserved for people applies to today's agents, because they already take part in processes with real impact. Starting now is simpler than adapting later.

Is it enough to have one person approving the AI's answers? Only if that person has enough evidence, time to read and the authority to refuse. Bulk approval without context becomes a formality and does not reduce risk. The process design needs to make review workable, with a clear summary, the applicable policy and escalation triggers.

Does the LGPD require human review of automated decisions? The LGPD gives data subjects the right to request review of decisions made solely on the basis of automated processing that affect their interests. How this applies to your case depends on the context and deserves legal assessment. In any case, keeping a contestation channel and a named owner is already good practice.

Where should a small company start? Pick a single flow, such as handling cancellations or triaging orders, and write on one page what the AI can do, what stays with a person and how the decision is recorded. Run that flow as a pilot, with a success criterion agreed in advance, and review the records before expanding.

Sources

• NIST: AI Risk Management Framework
• OECD: AI Principles
• ANPD: Autoridade Nacional de Proteção de Dados
• OWASP: Authorization Cheat Sheet