Ethics of Superintelligence for Business Leaders: Questions to Ask

The ethics of superintelligence, for business leaders, comes down to a set of governance questions: who answers for what the system does, what the company can explain and audit, and what limits exist before the system gets more autonomy. Superintelligence in the strict sense is still a hypothesis, but decisions about increasingly capable AI are already on the table, and the same questions apply to both cases.

Key takeaways

• Superintelligence is a hypothesis. What your company operates are models, agents and automations, and ethics starts with them.
• Accountability needs a name and a title: every system in production has an executive owner, and the model vendor does not fill that role.
• Transparency is the ability to reconstruct what the system did, with which data and who approved it. It is not a polished policy document.
• Set limits before granting autonomy: minimum permissions, human approval for irreversible actions and clear criteria for stopping.
• The more capable the system, the shorter the leadership review cycle should be, with a pilot, evidence and an explicit decision to expand, adjust or end it.

Is superintelligence a topic for today or for a distant future?

In the literature, superintelligence is a system that outperforms humans at most relevant cognitive tasks. No company operates this, and anyone selling it as if they do is doing marketing. What exists in real operations are language models, agents that call tools and automations with varying degrees of autonomy. The differences between these concepts are covered in the difference between AGI, ASI and AI agents, and it is worth reading superintelligence myths for business leaders before taking the topic to your board.

Why discuss ethics now, then? Because the cost of having no limits grows with the capability of the system. A leadership team that already knows who approves, who audits and who can switch off a simple agent is better prepared for more capable systems. If you cannot answer those questions for a customer support agent, you will not be able to answer them for something larger.

One reading precaution: when you consume material on this subject, separate three layers. The first is official documentation from vendors and reference bodies. The second is implementation experience, which describes what works in practice and depends on context. The third is hypothesis, including long-term scenarios. Mixing the three is the shortest path to a decision driven by fear or by enthusiasm.

Who is accountable when the AI gets it wrong?

Accountability cannot be delegated to the algorithm or to the model vendor. Brazil's data protection law (LGPD) distinguishes controller and operator and assigns duties to each, and a vendor contract does not remove the company's obligation toward the data subject. Confirm with your legal team how those roles apply to each tool. Questions leadership can ask:

• Who is the executive owner of each AI system in production, by name and title?
• Who approves changes to the model, the instructions and the permissions?
• Does the vendor contract say what happens to data, to failures and to incidents?
• Is there a channel for a customer or an employee to contest an answer or a decision?
• Who communicates an incident, to whom, and within what internal deadline?

If any answer is "it depends" or "the IT team", accountability is not yet defined. Technology operates the system, but the decision to put it in contact with customers, money or personal data belongs to the business.

What does the company need to be able to explain?

In practice, transparency means being able to reconstruct an episode: what input the system received, which data it consulted, what action it took, which version of the instructions and the model was active, and who approved it when approval was involved. This depends on decision logs and observability from the start. The observability primer from OpenTelemetry helps the technical team structure those records.

There is also the data subject's side. When an automated decision affects a person's interests, the LGPD provides a right to request review. The article LGPD automated decision-making and the right to review explains what this means for credit, collections and support flows. Useful questions for leadership:

• Can we explain to a customer, in plain language, how their answer was generated?
• Do the logs show the decision and also the context that led to it?
• Does the customer know they are talking to an AI when that matters?
• How long do we keep the logs, and who can consult them?
• Does data leave the country at any point in the flow, and does the vendor say where it is processed?

The last question has its own implications under the LGPD, covered in international data transfer when using AI providers.

What limits should the system have, and who can change them?

A limit is a design decision, not a promise from the model. Instructions written in natural language guide behavior, but they do not replace technical permissions. An agent with no access to the refund module cannot issue a refund, no matter how persuasive the person talking to it. This is the principle of least privilege, described in guides such as the OWASP Authorization Cheat Sheet, and it applies to people and to agents alike. The OWASP Top 10 for LLM applications includes the risk of excessive agency, which is exactly too much power given to the system.

A decision guide for limits:

• Reversible, low-impact actions, such as answering a question about opening hours, can be autonomous, with logging.
• Actions with financial, contractual or legal effect require human approval before execution.
• Irreversible actions, such as deleting data or sending a mass communication, require approval and a second control point.
• Sensitive data categories stay out of the agent's reach, unless leadership formally decides otherwise with a defined legal basis.
• Any authorized person must be able to pause the system without depending on the vendor.

And who can change these limits? If the answer is "whoever has access to the tool", a change process is missing. Changing instructions, tools or the model should produce a record and pass a test before going to production.

Illustrative scenario: the agent that starts negotiating

Illustrative scenario, not related to any real company. A services company uses a WhatsApp agent for questions and appointments. Because it works well, someone proposes that it start offering discounts and approving refunds to shorten the team's queue.

Instead of approving or vetoing right away, leadership asks the questions in this article. Who answers for the concessions the agent makes? The owner of customer service, with a registered name. What must be recorded? The customer's request, the rule applied, the amount granted and the version of the instructions. What is the limit? The agent can propose, but any concession above a ceiling set by leadership goes to a person. How does the customer contest? Through a clear path to a human. And how do we know it worked? With a criterion agreed in advance: for example, comparing complaints and rework before and after, on a restricted flow.

The outcome of that conversation is neither a refusal nor an open yes. It is a pilot with a small scope, technical limits and a date to decide the next step. For the channel itself, it helps to know the platform rules, such as those for outbound messages with templates and opt-in.

How to make this routine: map, limit, pilot, transfer

Questions only help if they become process. The method we apply in AI projects follows four moves: map the current process and the starting point, limit data and actions, pilot a real flow with logs and an agreed success metric, and transfer the operation, then decide whether to expand, adjust or end it. None of this requires philosophy, only management discipline.

• Map: describe the process as it is now, who decides what and where the data lives. Without this picture, there is no way to measure improvement or risk.
• Limit: define which data the system sees, which actions it executes and which require human approval. Write the shutdown plan.
• Pilot: choose one flow, run it with complete logs and evaluate it with test cases and human review. Evaluation guides such as OpenAI Evals show how to structure this.
• Transfer: hand over the operation with named owners, a review routine and stop criteria. Only then discuss widening the scope.

To structure outside or internal support for this process, the article on how Juan Carlo works with companies describes the format of diagnosis, pilot and handover.

When should leadership expand, adjust or stop?

The worst decision is the one never made: the pilot keeps running with no owner and no date. Define in advance what each outcome means.

• Expand: the agreed criterion was met, the logs are complete, there was no unanswered incident and the process owner wants to widen the scope.
• Adjust: the result is promising, but there are recurring failures in specific cases. Fix the instructions, tools or scope and repeat the test.
• Stop: the system acts outside its limits, the logs do not allow episodes to be reconstructed, the vendor does not explain how data is handled or nobody takes responsibility.

As external references, the NIST AI Risk Management Framework organizes AI risk management into governance, mapping, measurement and management functions, and the OECD AI Principles address transparency, robustness and accountability. It is worth cross-checking them with guidance from Brazil's data protection authority (ANPD) and following what is defined in the Brazil AI regulatory framework. These documents do not replace your company's legal opinion, but they give leadership and the technical team a shared vocabulary.

Frequently asked questions

Does leadership need an AI ethics committee? Not always. In smaller companies, one executive owner, a periodic review routine and a record of decisions are enough. A committee starts to make sense when there are several systems, sensitive data or decisions that significantly affect people.

Should we worry about superintelligence if we only use ChatGPT and AI on WhatsApp? The practical concern is autonomy, data and accountability, which already exist in those uses. The superintelligence debate helps train your reasoning, but the immediate work is a usage policy, limits and logs. The article on ChatGPT and superintelligence helps calibrate expectations.

What should we do when the vendor does not explain how the model works? Ask for what is verifiable: where data is processed, whether it is used for training, which logs are available and how incidents are communicated. You do not need to understand every internal detail of the model, but you need evidence about what goes in, what comes out and what is stored. If even that is not offered, treat it as a signal to limit use or change vendors.

What is the first question leadership should ask? Who answers for this system, and how can it be paused? If the answer is clear, the other questions about data, transparency and limits have something to rest on. If not, the first step is to name that owner before any expansion.

Sources

• NIST: AI Risk Management Framework
• OECD: AI Principles
• OWASP: Top 10 for LLM Applications
• ANPD: Autoridade Nacional de Proteção de Dados