Bookmarks (714)

Published on March 14, 2025 6:38 PM GMTRapid AI progress is the greatest driver of existential risk in the world today. But — if handled correctly — it could also empower humanity to face these challenges.Executive summary1. Some AI applications will be powerful tools for navigating existential risksThree clusters of applications are especially promising:Epistemic applications to help us anticipate and plan for emerging challengese.g. high-quality AI...

Published on March 14, 2025 3:00 PM GMT(Audio version here (read by the author), or search for "Joe Carlsmith Audio" on your podcast app. This is the fourth essay in a series that I’m calling “How do we solve the alignment problem?”. I’m hoping that the individual essays can be read fairly well on their own, but see this introduction for a summary of the essays...

Published on March 14, 2025 12:30 PM GMTDan Hendrycks, Eric Schmidt and Alexandr Wang released an extensive paper titled Superintelligence Strategy. There is also an op-ed in Time that summarizes. The major AI labs expect superintelligence to arrive soon. They might be wrong about that, but at minimum we need to take the possibility seriously. At a minimum, the possibility of imminent superintelligence will...

Published on March 14, 2025 9:48 AM GMTEpistemic status: somewhat rushed out in advance of the deadline tomorrow (Saturday the 15th) for the call for public comment on US AI policy. I think the issue is complex and deserves careful consideration.SummaryMethods and motivations for governments to control AGI and limit its proliferation seem to be underexploredCurrent laws appear adequate to take control of AGI...

Published on March 14, 2025 8:27 AM GMTA short science fiction story illustrating that if we fail to solve alignment, humanity risks losing not only 8 billion lives. He opened his eyes. The room was plain white."You remember enough?" asked the familiar voice."Enough," he replied. He stood. Everything balanced.He walked carefully down a bright hallway. At the end, a window showed leaves falling from a...

Published on March 14, 2025 4:52 AM GMTThis is a cross-post  - as some plots are meant to be viewed larger than LW will render them (and on a dark background) it is recommended this post be read via the original site.Thanks to Zach Furman for discussion and ideas and to Daniel Murfet, Dmitry Vaintrob, and Jesse Hoogland for feedback on a draft of...

Published on March 14, 2025 2:10 AM GMT While in a more remote area bike lights let you see where you're going, in a city there's usually enough light around for that. But you still need lights, so other people see you. [1] Unfortunately, lots of people end up biking without lights: it's easy to forget them, have them break, or end up out...

Published on March 13, 2025 8:36 PM GMTWhen communicating about existential risks from AI misalignment, is it more important to focus on policymakers/experts/other influential decisionmakers or to try to get the public at large to care about this issue?[1] I lean towards it being overall more important to communicate to policymakers/experts rather than the public. However, it may be valuable for certain individuals/groups to focus...

Published on March 13, 2025 7:18 PM GMTWe study alignment audits—systematic investigations into whether an AI is pursuing hidden objectives—by training a model with a hidden misaligned objective and asking teams of blinded researchers to investigate it.This paper was a collaboration between the Anthropic Alignment Science and Interpretability teams.AbstractWe study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As...

Published on March 13, 2025 6:29 PM GMTWe (SaferAI) propose a risk management framework which we think should improve substantially upon existing Frontier Safety Frameworks if followed. It introduces and borrows a range of practice and concepts from other areas of risk management to introduce conceptual clarity and generalize some early intuitions that the field of AI safety independently came up with.To maintain readability...

Published on March 13, 2025 6:17 PM GMTWhy do adults pursue long-term and complex goals? People don’t come into the world wanting to amass followers on Instagram, get a Ph.D., or become rich, yet these and a myriad of other goals are what seem to motivate most people. I have been interested in how these complex and long-term goals develop, and I think it...

Published on March 13, 2025 6:16 PM GMTThis post is a distillation of a recent work in AI-assisted human coordination from Google DeepMind. The paper has received some press attention, and anecdotally, it has become the de-facto example that people bring up of AI used to improve group discussions. Since this work represents a particular perspective/bet on how advanced AI could help improve human coordination, the following explainer...

Published on March 13, 2025 10:24 AM GMTI scrutinise the so-called "reversal curse", wherein LLMs seem not consier inverse relationships between conceptual nodes.I show that, far from being a proof of a lack of logical skills, it is a normal artefact of saliency, known in humans as associative recall asymmetry, and propose a conceptual-network model of the causes which works independently of substrate.Discuss

Published on March 13, 2025 2:40 PM GMTThe most hyped event of the week, by far, was the Manus Marketing Madness. Manus wasn’t entirely hype, but there was very little there there in that Claude wrapper. Whereas here in America, OpenAI dropped an entire suite of tools for making AI agents, and previewed a new internal model making advances in creative writing. Also they...

Published on March 13, 2025 12:58 PM GMTIf there is an international project to build artificial general intelligence (“AGI”), how should it be designed? Existing scholarship has looked to historical models for inspiration, often suggesting the Manhattan Project or CERN as the closest analogues. But AGI is a fundamentally general-purpose technology, and is likely to be used primarily for commercial purposes rather than military...

Published on March 13, 2025 12:08 PM GMTIntroSo far we have identified lock-in risk, defined lock-in, and established threat models for particularly undesirable lock-ins. Now we present this evaluation benchmark for large language models (LLMs) so we can measure (or at least, get a proxy measure for) the risk level of LLMs.AI is the key technology in the manifestation of lock-in risks; AI systems...

Published on March 13, 2025 10:52 AM GMTThis started life as a reaction to a post by Raemon, in which he calls on Anthropic to be more willing to halt its research if it enters a danger zone, and for them to take the relevance of technical philosophy much more seriously. My personal opinion is that at this point, even slowing things down (I mean...

Published on March 13, 2025 9:40 AM GMTCrossposted on the EA Forum.Displacement of other Space-Faring Civilizations (SFCs) may have macrostrategic impact for longtermists as hinted in a previous post. For example, our causal impact depends on how much value we produce relative to how much value would have been produced without us. In this post, we clarify the concept of Civ-Saturation, the metrics to...

Published on March 11, 2025 5:45 PM GMTEpistemic status: Speculative pattern-matching based on public information. In 2023, Gwern published an excellent analysis suggesting Elon Musk exhibits behavioral patterns consistent with bipolar II disorder. The evidence was compelling: cycles of intense productivity followed by periods of withdrawal, risk-taking behavior (like crashing an uninsured McLaren), reduced sleep requirements during "up" phases, and self-reported "great highs, terrible lows."Gwern's analysis...

Published on March 11, 2025 3:57 PM GMTTL;DR Large language models have demonstrated an emergent ability to write code, but this ability requires an internal representation of program semantics that is little understood. Recent interpretability work has demonstrated that it is possible to extract internal representations of natural language concepts, raising the possibility that similar techniques could be used to extract program semantics concepts....

Published on March 11, 2025 3:39 PM GMTForethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar.We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window.More details on our website.Why...

Published on March 11, 2025 3:38 PM GMTThis is a linkpost for a new paper called Preparing for the Intelligence Explosion, by Will MacAskill and Fin Moorhouse. It sets the high-level agenda for the sort of work that Forethought is likely to focus on.Some of the areas in the paper that we expect to be of most interest to EA Forum or LessWrong readers are:Section 3...

Published on March 11, 2025 2:30 PM GMTEpistemic status: The following isn't an airtight argument, but mostly a guess how things play out.Consider two broad possibilities:I. In worlds where we are doing reasonably well on alignment, AI control agenda does not have much impact.II. In worlds where we are failing at alignment, AI control may primarily shift probability mass away from "moderately large warning...

Published on March 11, 2025 1:35 PM GMTThis is a response to Matt's earlier post. If you see "a large mixture of alignment proxies" when you look at a standard loss function, my post might save you from drawing silly conclusions from the earlier post. If you parse the world into non-overlapping magisteria of "validation losses" and "training losses", then you should skim this...

Published on March 11, 2025 11:52 AM GMTTL;DR: We provide some evidence that Claude 3.7 Sonnet doesn’t encode hidden reasoning in its scratchpad by showing that training it to use paraphrased versions of the scratchpads does not degrade performance.The scratchpads from reasoning models look human understandable: when reasoning about a math problem, reasoning models consider intermediate steps similar to the ones I would use,...

Published on March 11, 2025 5:50 AM GMTThose engaged with questions of how to make the world a better place are probably all too familiar with the infighting that occurs among people who are on the same side of a political issue because of differences in methodology.  Among those concerned about AI x-risk, for example, PauseAI activists sometimes criticize LessWrong and EA participants for...

Published on March 11, 2025 4:56 AM GMTCognitive reframing is a powerful psychological technique that encourages subtle shifts in perspective, which can result in more positive and empowering interpretations to help challenge everyday negative thoughts and behaviors.First I’ll tell the story of how I accidentally discovered cognitive reframing on my own, then I’ll define what the technique is, state why it’s useful (particularly for...

Published on March 11, 2025 3:14 AM GMTYou learn the rules as soon as you’re old enough to speak. Don’t talk to jabberjays. You recite them as soon as you wake up every morning. Keep your eyes off screensnakes. Your mother chooses a dozen to quiz you on each day before you’re allowed lunch. Glitchers aren’t human any more; if you see one, run....

Published on March 10, 2025 6:06 PM GMTI just saw on twitter someone claiming that we "don't know how to raise the birth rate". No need to link them, it's a fairly common sentiment.Have you actually tried raising the birth rate? For real, as in actually put in any effort?Let's run the numbers on the stupidest possible thing that could work. Offer anyone who...

Published on March 10, 2025 4:07 PM GMTProduced as part of the ML Alignment & Theory Scholars Program -  Winter 2024-25 Cohort**Epistemic status**: I received positive feedback when I presented this idea to people at EAG, and my MATS mentor Evan Hubinger suggested I make a more detailed write-up to get feedback from the community. This proposal is at the theoretical stage with planned...

Published on March 10, 2025 3:35 PM GMTNote: This post is intended to be the first in a broader series of posts about the difficult tradeoffs inherent in public access to powerful open source models. While this post highlights some dangers of open models and discusses the possibility of global regulation, I am not, in general, against open source AI, or supportive of regulation...

Published on March 10, 2025 10:22 AM GMTEpistemic status: a combination and synthesis of others' work, analysed and written over a few weeks. A high-level list of threat models that is open to criticism.TL;DRHumanity could end up in a lock-in within the next century. Here I outline the possible routes to that outcome, and prioritise these routes on a set of criteria for importance.Existing...

Published on March 10, 2025 6:50 AM GMTAfter years of clumsily trying to pick up neuroscience by osmosis from papers, I finally got myself a real book — Affective Neuroscience, by Jaak Panksepp, published in 1998, about the neuroscience of emotions in the (animal and human) brain.What surprised me at first was how controversial it apparently was, at the time, to study this subject...

Published on March 10, 2025 1:26 AM GMTrelevant roon Our new friend in the cloud As I am writing this, I'm having a conversation with Claude about US bandwidth regulation while a trio of college students across from me attempt to wrangle an assignment from ChatGPT. Both of us, in our own ways, are talking to an oracle we trust as a matter of...

Published on March 9, 2025 9:10 PM GMT"Short AI timelines" have recently become mainstream.  One now routinely hears the claim that somewhere in the 2026-2028 interval, we'll have AI systems that outperform humans in basically every respect.For example, the official line from Anthropic holds that "powerful AI" will likely arrive in late 2026 or in 2027.  Anthropic's OSTP submission (3/6/2025) says (emphasis in original):[1]Based...

Published on March 9, 2025 6:09 PM GMTThis video provides a lot of background: https://www.youtube.com/watch?v=Eq3bUFgEcb4 and is also very funny, but it's not necessary to understand this post. If you've watched it you can probably skip to the "Why is Piano Roll Notation Bad?" section.TL;DRMusic notation is very janky and weird, but attempts to reform it to be "logical" are usually even worse. The...

Published on March 9, 2025 11:53 AM GMTPreserving the memory, and the cells, of the best cat everThis story begins in May 2007. The Iraq War was in full swing, Windows Vista was freshly released,[1] and I was just 10 years old.My family’s beloved black cat, Lucy, had died. Devastated, I begged my parents: we need to get another Lucy. They must have felt the...

Published on March 9, 2025 5:29 AM GMTI'm curious as to how well Claude can write interesting coding and mathematics problems. This post is a partial product of that exploration. It is a much harder skill to come up with a good problem than to solve one of equal difficulty. While there is a large corpus of data on solving problems, there is very little...

Published on March 8, 2025 4:40 PM GMTI'm in a coworking space on the 25th floor of a building in the bay. In the corner of my eyes, the red figure of the golden gate shimmers. Good morning America. I look down and see the people on the street walking into their offices, cafés, and autonomous Waymos. I stare onto San Francisco and feel...

Published on March 8, 2025 2:59 AM GMTKey points:Standardized assessments do not provide signals for the top ~10% of students.Education studies based on these poor signals lead to poor policies. For example, suspensions allegedly do not improve achievement among peers. However, the signal of "achievement" is "achieving credit in the class", which >80% of students already do. Thus, the conclusion is drawn from a...

Published on March 8, 2025 12:58 AM GMTAfter recently playing some chess against GPT-4.5 (it is pretty good, a lot stronger at the game than GPT-4 was - I would say strong club player level), I decided to try losing chess. To my surprise, it was able to play a complete game, which is something I have not seen any other model being able...

Published on March 7, 2025 8:09 PM GMTTheir comment was a paranoid and conspiratorial thought process about online tracking and surveillance. I saw it on Reddit.They start by questioning whether responding to the question would leave a record, proving they possess restricted knowledge. They continue down a rabbit hole, worrying about Reddit tracking their activity, considering private communication via .onion chatrooms, and realizing that...

Published on March 7, 2025 10:58 PM GMTIt's probably not possible to have a twin of me that does everything the same except experiences no qualia, i.e. you can predict 100% accurately, if you expose it to stimulus X and it does Y, that I would also do Y if I was exposed to stimulus X.But can you make an "almost-p-zombie"? A copy of...

Published on March 7, 2025 9:50 PM GMTLoosely inspired by a submission to a hackathon on Autostructures, which is about radical transformations from even mildly intelligent/agentic AI.Previously, I've briefly alluded to 6 infrastructural pillars for an era where "things that don't scale" start to scale. This post is mainly aimed at conveying an idea of the 1st pillar ("live interfaces"). The hope from interface...

Published on March 7, 2025 9:29 PM GMTIntroductionRecently, the Computational No-Coincidende Conjecture[1] was proposed, presented as an assumption that might be needed to develop methods to explain neural nets in the current approach from the Alignment Research Center. In this post, I will prove that some plausible extra assumption can be used to amplify the conjecture, showing it implies an arbitrarily stronger version of itself.In...

Published on March 7, 2025 7:49 PM GMTTL;DR: PAIR on AI & Reasoning. ESPR on Everything, including AI and Reasoning. If you are 16-21 yo and are interested in AI, Rationality or Everything, we encourage you to apply by March 16th.The FABRIC team is running two immersive summer workshops for mathematically talented students this year.The Program on AI and Reasoning (PAIR) for mathematically talented students who...

Published on March 7, 2025 6:17 PM GMTHighlights:Manifold ending (a) cash markets, Kalshi slapped by regulators in NevadaInteractive Brokers now also offering election markets (a)Violet Hour looks at Where Would Good Forecasts Most Help AI Governance Efforts? (a).Discuss

Published on March 7, 2025 12:40 PM GMTThis complication of tales from the world of school isn’t all negative. I don’t want to overstate the problem. School is not hell for every child all the time. Learning occasionally happens. There are great teachers and classes, and so on. Some kids really enjoy it. School is, however, hell for many of the students quite a...