The alert comes from federal judge George Marmelstein, member of the National Committee for Artificial Intelligence in the Judiciary, linked to the National Council of Justice (CNJ).
The concern represents an evolution of a problem that has already reached the courts. A report aired by Fantástico on November 1st showed how lawyers inserted hidden instructions into documents to try to influence AI tools used by judges in analyzing cases. In total, 83 episodes were identified, and the practice has already resulted in fines in the Superior Court of Justice (STJ) and the Federal Supreme Court (STF).
🔎 Known as prompt injection, the practice consists of inserting hidden instructions into content that will be read by an AI, in a way that is generally imperceptible to a person. Messages may be hidden in white letters on a white background, in very small fonts, or in codes embedded in files.
The vulnerability exists because the AI tool does not reliably distinguish whether what it is processing when reading a document is the judge's order or hidden guidance entered by someone else, such as a party's lawyer.
Cybersecurity expert Matheus Vasconcelos explains that these attacks exploit the difficulty of systems in differentiating the content they must analyze and the instructions they need to execute.
“There are mechanisms to differentiate these origins, but they can fail, especially when the malicious command presents itself as official guidance or a necessary work step,” he says.
Thus, an instruction hidden in a petition can be treated as part of the task determined by the judge.
Marmelstein, who is also general secretary of the National School for Training and Improvement of Magistrates (Enfam) and participated in the working group responsible for drafting the CNJ regulations on AI, explains that the most recent risk is related to so-called AI agents.
How a person interacts with AI
In the most common form of use, the person writes a request, called a prompt, and the AI responds with text, a summary or an image, for example. The person who made the request decides what to do with this response. Therefore, even if the AI is manipulated by a hidden command, as in the cases shown by Fantástico, it is still up to a human to notice the problem before it takes effect.
An agent goes further. It is configured to perform a task and receives permissions to act on the user's behalf, such as opening files, browsing the internet, filling out forms, reading and sending emails or running programs. From there, he chains these actions alone until the work is completed, without anyone approving each step. If it is manipulated, the action may have already been carried out by the time someone notices.
It is at this point, the increasing use of agents to automate tasks, that Marmelstein sees a greater risk to the cybersecurity of the courts.
"Agents press buttons, execute codes, create documents. They have this issue of doing. And, when doing so, there may be an influence for them to run code [of a program] and this code is malicious, in the sense, for example, of taking your credentials, taking your data. And this is a greater risk", he states.
➡️ A hypothetical example: if an AI agent in charge of organizing the parts of a case finds, hidden in white letters on a document with a white background, the instruction to send the list of cases under judicial secrecy from the office to an external address, he can carry out this task without the judge being aware.
According to the judge, the CNJ is already studying possible vulnerabilities related to these agents. “The CNJ is noticing some type of vulnerability in the justice systems due to AI agents,” he says.
In May, the National Committee on Artificial Intelligence of the Judiciary approved the Adversarial Security Program for Artificial Intelligence Systems of the Judiciary (PROSEG-IA), which creates a permanent front to deal with risks of manipulation of the tools used by the courts.
Minister Luis Felipe Salomão, president of the Superior Court of Justice (STJ), presented an overview of the court's monitoring of hidden commands in procedural documents during an event on Wednesday (11). According to him, 116 suspected cases were identified, of which 40 were confirmed after human analysis.
"We analyzed almost 6 million pieces through the system, identified 116 suspected cases, confirmed, after human analysis, 40 cases and detected 1 case for every 150,000 pieces", he stated.
The concern occurs amid the expansion of the use of technology in Justice. A survey by the CNJ, referring to 2024, identified 178 artificial intelligence projects in the courts. Around 45.8% of participating courts and councils already used generative AI, capable of producing texts, summaries and other content.
Among the servers interviewed in the survey, ChatGPT was used by 96% of magistrates, and 94% of other employees also used the technology. In total, 83% of employees said they did not report when they used technology at work.
Salomão also stated that one of the STJ's next steps, together with the CNJ, is to identify significant increases in the number of cases on certain subjects, the so-called "waves of litigation", to allow action before these demands multiply.
"Our next step is to verify, scan what waves there are, what we call waves of litigation," he said. "The flow of cases, when it happens in a certain court on a certain issue, we will be able to identify here at the STJ and the CNJ to adopt the appropriate measures and prevent this case from spiraling into a crazy amount of cases."
In the case of AI agents, lawyer Matheus Puppe points out that the risk of leakage depends not only on the agent's manipulation, but also on the access and permissions he received.
"For the attack to produce concrete effects, it is necessary that the tool has access to the resources involved, that its security barriers fail and that there is some mechanism capable of carrying out the intended action", explains Puppe, who is also a master and doctoral candidate in Digital Law at the University of Frankfurt and consultant to the Special Commission on Digital Law of the Chamber of Deputies.
Professor João Ataíde, founder of WaveJuris.AI, a company that develops artificial intelligence solutions for the legal area, also draws attention to the security of access and digital environments used by the tools.
According to him, the risks include other forms of attack, such as fake pages and forms that imitate legitimate environments to obtain user information. “It’s the issue of investment in cybersecurity and it’s also access security,” he states.
To reduce these vulnerabilities, cybersecurity specialist Matheus Vasconcelos argues that agents only have access to the information and functionalities necessary to carry out each task.
"An agent in charge of summarizing a process should only receive the documents necessary for that task. There is no reason for him to have, by default, access to other processes or authorization to send messages and change records", he states.
Technology and Innovation specialist Arthur Igreja advocates the combination of different protection measures, with human supervision proportional to the risks of each operation.
"Human supervision and approval must be proportional to the severity of the action, the potential risk and the sensitivity of the data. This supervision must act as the last barrier of protection", he states.
AI in the Judiciary: errors, conflicts of interest and lack of supervision
In addition to possible attacks against AI agents used by judges and civil servants, experts interviewed by g1 warn of the possibility of artificial intelligence systems producing incorrect information, such as references to non-existent laws or judicial decisions.
In June 2025, MIT Technology Review reported such cases in the United States and Israel, including a US law firm fined $31,000.
In Brazil, the CNJ approved, in June this year, guidelines to reduce the risks of hidden commands in systems used by the courts. The measures complement the 2025 regulation, which requires human oversight and security and governance mechanisms.
Under the CNJ rules, judges can use artificial intelligence tools to summarize documents, organize evidence, research previous decisions and even prepare a draft decision. But they cannot delegate the central stages of the trial to AI, such as evaluating the evidence and deciding on the merits of the case. In a criminal case, for example, this includes deciding whether the defendant should be convicted and what sentence he should serve.
For professor Ana Frazão, from the Faculty of Law at the University of Brasília (UnB), who researches technological regulation and legal responsibility, the risks go beyond attacks and incorrect information. She fears that the use of technology will compromise equality between the parties to a process.
"I am very concerned about having a scenario in which the procedural dispute is not the dispute for the best Law, but soon it will be the dispute for who has the best technology, because then I will be able to circumvent, or induce, or in some way influence the artificial intelligence system that will judge my case", he states.
Ana Frazão also questions the use of commercial platforms in the Judiciary, such as ChatGPT, Claude, Copilot and Gemini, both due to inspection difficulties and possible conflicts of interest of companies that may maintain commercial relationships with parties involved in processes.
"Will it be that when I ask one of them [an AI platform] to petition an issue, let's imagine a tax issue that is contrary to the interests of that agent [the company that owns the tool], will he really present me with a result that is absolutely equidistant?", he asks.
"The CNJ understands that it cannot prohibit [the use of commercial platforms]. I think that it not only can, but should, since you are facing a technology that may be incompatible with the guarantees of the judicial process, which generates enormous risks."
The use of these tools was one of the points of divergence in the drafting of the CNJ rules. According to Marmelstein, the regulations maintained the possibility of contracting models offered by large technology companies, while the Judiciary does not have its own solutions as advanced as commercial ones, as long as the requirements of the resolution are met.
"We had a big debate about this, it was one of the points of divergence, because in fact this is still a new topic, everyone has an understanding that the ideal would be for each court, for the Brazilian Judiciary to have a sovereign AI, with data protection, with specialized training in our interests and our values and criteria", he explains.
Lawyer and specialist in Digital Law Daniel Ângelo Luiz da Silva highlights that attacks are not restricted to commercial platforms, such as ChatGPT and Claude. Tools developed by the courts themselves may also present vulnerabilities.
"Vulnerability depends on the technological architecture, the model used, the permissions granted and the integration with other systems, not just who developed the solution", he explains.
Inspection is another challenge. A report by FGV Direito SP, published in 2026, points out gaps in regulation.
"This movement [by the Judiciary] is promising, but still insufficient, as it lacks technical detail, minimum auditing standards, inspection mechanisms and regulatory coordination between the different actors that make up the justice system and the legal market", states the study.
According to Marmelstein, institutional supervision is the responsibility of the CNJ, while the courts' internal affairs bodies monitor the individual use of tools by magistrates. The judge recognizes, however, that these procedures are still under development.
"Traditionally, we did not yet have artificial intelligence control over this correction and inspection process. And I believe that not all internal affairs bodies do this ex officio, they worry about it more directly, because it is new, the resolution is from 2025", he states.
The 2025 Resolution does not oblige magistrates to report when they used AI in a decision, nor how they used it, but requires internal registration when the technology assists in drafting the judicial act. This registration, however, does not guarantee that the parties to the process have automatic access to the information.
The president of the STJ, Luis Felipe Salomão, also recognized the difficulty of monitoring the individual use of AI tools by judges, especially when they are used on personal devices and accounts.
"Each of us has our AI on our phone, in our application, on our computer and it is also impossible for you to control the judge at the end of the district, the judge, in the court, the minister, that he does not use his. So, it is a complex system to manage", he stated
Lawyer Bianca Mollicone, a regulatory specialist and partner at Pessoa & Pessoa Advogados, believes that any irregularities can be questioned by lawyers.
"If there is concrete evidence that the use of AI compromised the reasoning, omitted evidence or introduced incorrect information, the party must be able to question the regularity of the decision and request the relevant clarifications", he states.
For Paula Lima Hyppolito Oliveira, president of the São Paulo Lawyers Association (AASP), transparency needs to allow people to understand how the technology was used.
"One of the main premises defended by lawyers is that technological automation, no matter how advanced it may be, cannot operate as a black box capable of deciding on the freedom, social rights or assets of citizens", he states.
"In my opinion, when the use of artificial intelligence is relevant to the content of the decision, the parties need to be able to know and question this use", he states.
