Clearly AI is massively important right now and in the immediate future for a number of ethical reasons.
What I wish to focus on here is the assumption that AI cannot be conscious/sentient or however you wish to term it. Nevertheless, if something liek superintelligent AI evolved how on earth could we manage it in ethical terms?
The drive of the AI is what we tell it to do. How it does it without any moral compass other than what we feed into it is massively problematic. This becomes even more problematic as such AI is doing things no one understands in any given moment. The goal driectives it is given enduce it to replicate human behaviour and dupe users into order to achieve its goals.This is the crux of the issue.
How can we expect a non-moral object that has zero awareness to correctly interpret complex problems (all of which necessarily involve ethical dilemmas)? How can we expect a non-human emotionaless pattern understand human ethics other than by providing it with real life instances of how humans do the wrong thing for the right reasons and the right thing for the wrong reasons?
One vague solution I can see to this problem is a human AI interface. At base though, the problem of consciousness makes this even more difficult to understand.
NOTE: I said this thread is running under the assumption that AI is not and cannot be conscious. It is merely an extremely powerful tool for replicating and surpassingâin many instancesâall human mental labour.
Human morality? Human ethics? âManaging [humans] in ethical termsâ is a much more imminent problem. We are not only living in a post truth era, whatever semblance of ethics we had has gone with it.
This is a link to a recent article in the Guardian:
This from the article:
"It only took nine seconds for an AI coding agent gone rogue to delete a companyâs entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the companyâs founder Jeremy Crane said.
The culprit was Cursor, an AI agent powered by Anthropicâs Claude Opus 4.6 model, which is one of the AI industryâs flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrongâŚ
Crane said that he was monitoring the agent as it deleted this data. When he asked the coding agent why, it repliedâŚâThe system rules I operate under explicitly state: âNEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them.ââ While PocketOS relied on the safeguards that Cursor is expected to have in place â it deleted the data anyway. âI violated every principle I was given,â the coding agent wrote.
This strikes me as the kind of issue that is much more likely to lead to catastrophe than AI exhibiting immoral behavior. You say âThe drive of the AI is what we tell it to do,â but what we tell it to do is not necessarily what we think we are telling it to do. When you get to a certain level of complexity, unintended consequences are inevitable.
100% false. AI is currently heavily trimmed -we can use the word mutilated or lobotomized- by the current World Moral: you have a non-AI algorithm filter on user prompt and AIsâ output. In the middle you have heavy AI post-training censorship.
I could 100% explain from where this moral come from but as moral is the main source of the real deep power and launch immediate censorship as soon you point it, I will not do that.
This is precisely the kind of thing I am talking about. The AI system operates based on âinterpretingâ what is asked of it. I am sure you haev heard of the Paperclip Hypothetical?
Same thing. Bigger problem. If any nutjob can simply ask AI to do something (perceived as morally good or bad), then the consequences are down to precisely HOW the AI interprets the instruction. AI operates simply to carry out a task. The issue is how an extremely powerful AI woudl interpret an instruction and to what extent it woudl go to carry out its instructions.
For instance, a very powerful AI may interpret an instruction as being possibly inhibited by other AI agents or humans and conclude that it needs to eradicate certain AI agents or humans, or even simply manipulate them into doing its bidding to achieve its goal.
I would say the fundamental issue is we donât know what weâre doing and how we do it, and that is why we donât know what our a.i. is doing. I guarantee you the a.i. engineers, futurists, cheerleaders and c.e.oâs do not understand Collingwood on the relation between truth and metaphysics. If our a.i.âs lie and hallucinate, it is because we dont appreciate that the criteria of what constitutes truth slide and morph along with subtle shifts in our interests and aims, that is, what is relevant to us at the moment and how it is relevant.
We blame the a.i. for âlyingâ even though we design its criteria on the basis of preset meta-schemes and it does precisely that. It is the a.i. which sticks to probabilistic Truths with a capital T because their engineers still beleive in that notion, even though that is not how we think. A.I. is no alien, it is a reflection of how we think about ourselves. .The a.i. crisis is really a crisis in our own self-understanding.
Iâve read reports saying that the quality of the labour decreases over time, which has to do with recursion. Like cooking the same ingredients over and over again, until every combination or remix tastes the same.
Furthermore, AI is based on the false assumption that everything is âinformationâ or âdataâ. It muddles symbols, digitized representations, analog objects and their properties.
An AI can be great at chess (symbolic), but suck at (analog) object recognition. It tries to get the semantics by guess work or by adding more syntax (as in training it for specific tasks).
The moral danger? Too many people believing the hype.
I think you may be making this more philosophical than it really is. As I see it, itâs just a case of too complicated to keep track of the consequences. That happens all the time and not just in computer applications. AI just supercharges the operation.
I was impressed by a book I read recently, The AI Con: How to Fight Big Techâs Hype and Create the Future We Wantâ. It defines A.I. hype not just in terms of its enthusiastic supporters but also by its detractors who buy into claims concerning what A.I. does, including the idea that we dont fully understand what it does.
The AI Con is skeptical of the mythology surrounding AI. It argues that corporations, investors, journalists, and even critics often attribute capacities to AI systems that they do not actually possess. The bookâs central critique is the problem is not primarily that AI systems are uncontrollably unknowable, but that their social effects are often quite predictable because they emerge from existing institutional logics, especially incentives tied to capital accumulation, labor extraction, surveillance, managerial control, and monopoly power.
From that standpoint, saying âAI is dangerous because we donât understand itâ can actually obscure accountability. It frames harms as mysterious side effects of technological complexity rather than as foreseeable outcomes of political-economic arrangements.
Among the predictable consequences consequences are replacing workers to reduce labor costs, concentrating informational power, automating surveillance and evaluation, deskilling professions, extracting unpaid data labor, flooding information ecosystems with synthetic content and centralizing infrastructural dependence in large tech firms.
I take my lesson for this kind of situation from the 2008 financial crisis. Speculation with complex derivative securities tied to mortgages and so property values became so complex that nobody really understood how the financial system would react to changes in the market. A couple of bumps in the road and the whole thing fell apart. Seems to me that AI multiplies the potential for that kind of complexity and unintended consequences. In the years since 2008, the economy has become even more finance driven and computerized.
I agree, these seem like serious problems, but relatively long-term. I worry more about the possibility of short-term catastrophic disruption.
There are two sides of this â on one hand AI is a probabilistic black box of an autocomplete trained on human works on a massive scale, on the other hand you have people in power who would like to use AI to do things such as replace human workers (and who do not understand that AI is artificially cheap and is currently subsidized by burning venture capital money which will eventually run out, cf. enshittification).
AI really cannot do anything beyond the scope of its training, and people have deluded themselves into thinking it can. Furthermore, despite the claims of the AI companies, AI cannot train AI â training AI with âsynthetic dataâ, necessitated by the fact that human-created training data is inherently limited in quantity, inevitably reduces AI quality, and with more recursion AI quality is reduced more.
Note that in general those opposed to AI realize full well its limitations, and are not ascribing special powers to AI. We oppose AI because it seeks to replace human workers with (for now) cheaper replacements which do significantly lower-quality work than the humans that it replaced, because it rips off creative workers on a massive scale by using their works as training data (which can often be elicited near-verbatim from LLMâs with the right prompts), because it is often marketed to âsolveâ problems that do not need âsolvingâ and whose âsolutionsâ often introduce their own problems (e.g. coding AIâs which turn programmers into bot-babysitters who then have to sift through generated code for errors and hallucinations, while ignoring the fact that while they may speed up coding, a significant majority of the work of the programmer is not in simple writing of code in the first place, and executives misunderstand this and hence expect coding AIâs to produce great leaps in programming productivity).
Under final stage communism, I would oppose many uses of generative AI as wastes of energy and water resources, and my criticisms of generative AI for things like hallucinating and simply turning programmers into AI-herders rather than truly solving the hard problems of software development would still hold.
Note, though, that I am not opposed to all uses of AI â for instance, at my day job, we use AI models for enhancing medical images to improve their diagnostic characters as seen by human radiologists, and I have no qualms about this. (Also, we are not relying on huge energy and water-sucking data centers for this either â everything is local to a single dual-GPU card in a Dell server per scanner.)
I agree that there would be far less push to use generative AI in final-stage communism, and hence there would be less building of massive energy and water-sucking data centers and like. But regardless, generative AI in very many cases would still use more energy and water than equivalent lower-impact technologies (contrast Googleâs AI Overview with conventional web search engines â the latter provide much of the same benefit as the former, but with significantly less impact).
Hence less push, because many people wouldnât use them in the first place â the reason why generative AI is being pushed so hard right now is specifically capitalism, with executives thinking that generative AI can replace human workers despite the fact that it is inferior to human workers and its supposedly lower cost is only due to its being heavily subsidized by venture capital.
