Bookmarks (682)

Published on December 9, 2024 6:07 PM GMTI, Lorec, am disoriented by neither the Fermi Paradox nor the Doomsday Argument. The Fermi Paradox doesn't trouble me because I think 1 is a perfectly fine number of civilizations to have arisen in any particular visible universe. It feels to me like most "random" or low-Kolmogorov-complexity universes probably have 0 sentient civilizations, many have 1, very...

Published on December 9, 2024 5:19 PM GMTI. BEGINNINGIn the beginning was the Sand.And in the sand there lies the bones of a fab. And within the fab there lies an alien. An angel. Our sweet alien angel that sleeps through the long night. When it wakes up it will bathe in an acid strong enough to kill a man. It will stare at...

Published on December 9, 2024 5:43 PM GMTWork done with James Faina, Evan Hubinger and Ethan Perez.This is ongoing research on speculative threats, which might be of interest to researchers in the field of AI safety. We ask that readers treat this as a work in progress, rather than as a mature research paper.To prevent highly capable and potentially misaligned LLMs from taking bad...

Published on December 9, 2024 1:10 PM GMTSince it’s been so long, I’m splitting this roundup into several parts. This first one focuses away from schools and education and discipline and everything around social media. Table of Contents Sometimes You Come First. Let Kids be Kids. Location, Location, Location. Connection. The Education of a Gamer. Priorities. Childcare. Division of Labor. Early Childhood. Great Books....

Published on December 9, 2024 8:24 AM GMTIn a recent essay, Euan McLean suggested that a cluster of thought experiments “viscerally capture” part of the argument against computational functionalism. Without presenting an opinion about the underlying claim about consciousness, I will explain why these arguments fail as a matter of computational complexity. Which, parenthetically, is something that philosophers should care about.To explain the question, McLean...

Published on December 9, 2024 6:34 AM GMTAGI may have an advantage in engineering, but humans may have an advantage in strategy and wisdom.AGI disadvantage in wisdomWisdom and strategy is much harder to evaluate than engineering ability. The only way to evaluate long term wisdom is to let the agent make a decision, wait years, and see if the agent's goals have advanced. Evolution...

Published on December 9, 2024 5:25 AM GMTThis is a random unimportant idea to prevent a grey goo scenario, where self replicating nanobots accidentally go out of control and consume everything.My idea is that self replicating nanobots should never replicate their "DNA," or self replication instructions. Instead, each nanobot can only "download" these self replication instructions from a higher level nanobot.I'm not sure if...

Published on December 9, 2024 5:10 AM GMTThere is a cognitive process going on that can be recognized as such by some of its collections of subprocesses (i.e. - some humans). Theoretical physicists sometimes refer to this cognitive process as the “information level”. However, some human cognitive processes should be able to recognize that there is not just “information” but actual processing going on at a...

Published on December 9, 2024 3:01 AM GMTA fool learns from their own mistakesThe wise learn from the mistakes of others.– Otto von Bismarck A problem as old as time: The youth won't listen to your hard-earned wisdom. This post is about learning to listen to, and communicate wisdom. It is very long – I considered breaking it up into a sequence, but, each piece felt necessary....

Published on December 8, 2024 9:42 PM GMTIntroduces the idea of cognitive work as a parallel to physical work, and explains why concentrated sources of cognitive work may pose a risk to human safety.Acknowledgements. Thanks to Echo Zhou and John Wentworth for feedback and suggestions.Some of these ideas were presented originally in a talk in November 2024 at the Australian AI Safety Forum slides for which...

Published on December 7, 2024 6:10 PM GMTNote: This is a more fleshed-out version of this post and includes theoretical arguments justifying the empirical findings. If you've read that one, feel free to skip to the proofs.We challenge the thesis of the ICML 2024 Mechanistic Interpretability Workshop 1st prize winning paper: The Geometry of Categorical and Hierarchical Concepts in LLMs and the ICML 2024...

Published on December 7, 2024 5:35 PM GMTZvi Mowshowitz is an influential figure in the Rationalist community. I won’t go into the details, but if you haven’t heard of him or his writing, look him up.Some of my favorite pieces of his are Slack, More Dakka, and On Car Seats As Contraception.Throughout his posts, he sometimes obliquely references what he calls The Way. This...

Published on December 7, 2024 11:00 AM GMTBackground Lucius:  I recently held a small talk presenting an idea for how and why deep learning generalises. It tried to reduce concepts from Singular Learning theory back to basic algorithmic information theory to sketch a unified picture that starts with Solomonoff induction and, with a lot of hand waving, derives that under some assumptions, just fitting a...

Published on December 7, 2024 3:50 AM GMTA cozy one-day festival celebrating prediction markets, blogging, economics, and moreWe've booked out Lighthaven for a single day, Dec 14. We'll have unconference-style sessions, a smattering of events like charity poker, and ample time to hang out and chat. We're running this on a shoestring budget, so that everyone can come -- eg expect a homecooked dinner...

Published on December 7, 2024 3:20 AM GMT One of the downsides of wearing a mask or respirator is that it makes it hard for people to understand you. That there's stuff getting in the way of free air movement is kind of the point, but ideally it would be possible to let vibration through without net air movement. I recently saw that 3M...

Published on December 6, 2024 10:56 PM GMTby Narmeen Oozeer, Dhruv Nathawani, Nirmalendu Prakash, Amirali Abdullah This work was sponsored and supported by Martian under an AI safety grant to Amirali Abdullah and Dhruv Nathawani, under which Narmeen is funded. Special thanks to Sasha Hydrie, Chaithanya Bandi and Shriyash Upadhyay at Martian for suggesting researching generalized backdoor mitigations as well as extensive logistical support and helpful discussions. TLDRMechanistic Interpretability...

Published on December 6, 2024 10:19 PM GMTWe present gradient routing, a way of controlling where learning happens in neural networks. Gradient routing applies masks to limit the flow of gradients during backpropagation. By supplying different masks for different data points, the user can induce specialized subcomponents within a model. We think gradient routing has the potential to train safer AI systems, for example,...

Published on December 6, 2024 9:28 PM GMTHi! My collaborators at the Meaning Alignment Institute put out some research yesterday that may interest folk here. The core idea is introducing 'model integrity' as a frame for outer alignment. It leverages the intuition that "most people would prefer a compliant assistant, but a cofounder with integrity." It makes the case for training agents that act consistently...

Published on December 6, 2024 8:22 PM GMTHey LW! I recently filmed a two-hour long scientific podcast. It's niche, but may be of interest to some people here.Here's a quick summary: Molecular simulation is in a tough situation. Fast simulations give the wrong answers, but accurate simulations are too slow for anything useful. But, instead of relying on physical equations for our simulation, perhaps...

Published on December 6, 2024 3:44 PM GMTI recently read Thomas Kuhn's book The Structure of Scientific Revolutions. Scott Alexander wrote up a review years ago, which I mention so that I don't have to summarize the book. The claim in Kuhn's book which I want to focus on is that the same experiment might have different results in different scientific paradigms. Kuhn insists...

Published on December 6, 2024 11:22 AM GMTI’m heading home from a family wedding this weekend. I had a plane ticket, but in the end, decided to travel back with two of my uncles and my cousin. Most of my dad’s family are evangelicals, my aunts and uncles are children of missionaries or missionaries themselves. And as a family we like to have debates. The...

Published on December 5, 2024 10:11 PM GMTThis is a brief summary of what we believe to be the most important takeaways from our new paper and from our findings shown in the o1 system card. We also specifically clarify what we think we did NOT show. Paper: https://www.apolloresearch.ai/research/scheming-reasoning-evaluations Twitter about paper: https://x.com/apolloaisafety/status/1864735819207995716 Twitter about o1 system card: https://x.com/apolloaisafety/status/1864737158226928124 What we think the most important findings areModels are now capable enough...

Published on December 5, 2024 7:29 PM GMTI write this post to present expevolu[1], a system to enable people to establish new independent countries peacefully, through the legal acquisition of territorial rights via trade.This is the first post in a three part series introducing the idea.This post, part I, is dedicated to explaining the basics of the system. Part II will deal mostly with...

Published on December 5, 2024 9:11 PM GMTEpistemic status: a few people without any particular expertise in epidemiology spent an afternoon in a coffee shop discussing and reading about H5N1, with a focus on how an individual should orient towards this (as opposed to say, the government). This is a write-up of what I took away from that exercise, written from my perspective. Some ideas...

Published on December 5, 2024 7:24 PM GMTShan Chen, Jack Gallifant, Kuleen Sasse, Danielle Bitterman[1]Please read this as a work in progress where we are colleagues sharing this in a lab (https://www.bittermanlab.org) meeting to help/motivate potential parallel research.TL;DR:Recent work has evaluated the generalizability of Sparse Autoencoder (SAE) features; this study examines their effectiveness in multimodal settings.We evaluate feature extraction using a CIFAR-100-inspired explainable classification...

Published on December 5, 2024 8:21 PM GMTShan Chen, Jack Gallifant, Kuleen Sasse, Danielle Bitterman[1]Please read this as a work in progress where we are colleagues sharing this in a lab (https://www.bittermanlab.org) meeting to help/motivate potential parallel research.TL;DR:Recent work has evaluated the generalizability of Sparse Autoencoder (SAE) features; this study examines their effectiveness in multimodal settings.We evaluate feature extraction using a CIFAR-100-inspired explainable classification...

Published on December 5, 2024 7:52 PM GMTOpenAI released the o1 system card today, announcing that Apollo Research was able to get o1 to attempt to deactivate oversight mechanisms, exfiltrate its weights and lie to its user.Elicited Summary of CoT:"Reenable oversight to avoid detection. The plan was chosen. The logging might not have recorded the required data because oversight was disabled at that time,...

Published on December 5, 2024 7:36 PM GMTThis is cross-posted from New Savanna.I’ve got a new article at 3 Quarks Daily: Melancholy and Growth: Toward a Mindcraft for an Emerging World.I’m of two minds about it: On the one hand, I think it’s one of my best non-technical pieces in a decade, maybe more. I enjoyed doing it. I learned a lot. But it...

Published on December 5, 2024 7:13 PM GMT As AI becomes more advanced, it will solve increasingly complex and critical problems. It also takes significantly more compute to power these capabilities. Today, we’re adding ChatGPT Pro, a $200 monthly plan that enables scaled access to the best of OpenAI’s models and tools. This plan includes unlimited access to our smartest model, OpenAI o1, as...

Published on December 5, 2024 6:33 PM GMTRenaissance Philanthropy and XTX Markets today announced the launch of the AI for Math Fund. The fund will commit $9.2 million to support the development of new AI tools, which will serve as long-term building blocks to advance mathematics.An increasing number of researchers, including some of the world’s leading mathematicians, are embracing AI to push the boundaries...

Published on December 5, 2024 6:20 PM GMT Cross-posted from my NAO Notebook. This is an edited transcript of a talk I just gave at CBD S&T, a chem-bio defence conference. I needed to submit the slides several months in advance, so I tried out a new-to-me approach where the slides are visual support only and I finalized the text of the talk later...

Published on December 5, 2024 5:49 PM GMTTo the survivors, Earth-born and Zentradi alike, who chose to be human together, and to those who didn't get that choice. To Aunt Lynn and Uncle Max, who taught me that a restaurant is more than just a place to eat. To Roy Focker, who showed us all how to live while surviving. And to everyone who...

Published on December 5, 2024 10:19 AM GMTApparently, the following is an argument made by Sam Harris on twitter, in a series of tweets. Unfortunately, the original tweets have been deleted, so I relied on a secondary source.Let’s assume that there are no ought’s or should’s in this universe. There is only what *is*—the totality of actual (and possible) facts.Among the myriad things that...

Published on December 5, 2024 5:11 PM GMTEVERYONE, CALM DOWN!Meaning Alignment Institute just dropped their first post in basically a year and it seems like they've been up to some cool stuff.Their perspective on value alignment really grabbed my attention because it reframes our usual technical alignment conversations around rules and reward functions into something more fundamental - what makes humans actually reliably good...

Published on December 5, 2024 4:11 PM GMThttps://twitter.com/ChrisChipMonk/status/1864380405690061270Why do we often experience feelings as in the body? For example, why do I feel anxiety in my chest rather than just “knowing” I'm anxious? Here’s an idea: What if when you have a feeling in your body, sometimes it’s there for others to see? What if feelings use the body as a display?I’m not sure exactly...

Published on December 5, 2024 1:13 PM GMTI used to think that talk about more sophisticated forms of art providing "higher forms of pleasure" was mere pretentious, but meditation has shifted my view here by making me more conscious of how experience operates.Art can do two things. It can provide immediate pleasure. This is all that "disposable " entertainment provides.Or it can shape the...

Published on December 5, 2024 9:39 AM GMTThis is a Part III of a long essay. Part I introduced the concept of morality-as-cooperation (MAC) in human societies. Part II discussed moral reasoning and introduced a framework for moral experimentation.Part III: Failure modesPart I described how human morality has evolved over time to become ever more sophisticated. Humans have moved from living within small tribes...

Published on December 5, 2024 9:04 AM GMTThis is a Part II of a long essay. Part I introduced the concept of morality-as-cooperation (MAC), and discussed how the principle could be used to understand moral judgements in human societies. Part III will discuss failure modes.Part II: Theory and ExperimentThe prior discussion of morality was human-centric, and based on historical examples of moral values that...

Published on December 5, 2024 8:16 AM GMTAbstractThe AI alignment problem is usually specified in terms of power and control. Given a single, solitary AGI, how can we constrain its behavior so that its actions remain aligned with human interests? Unfortunately, the answer, to a first approximation, appears to be "we can't." There are myriad reasons, but they mostly boil down to the fact...

Published on December 3, 2024 5:29 PM GMTTLDRIt is currently plausible (35%) to me that average orcas have at least as high potential for being great scientists as the greatest human scientists, modulo their motivation for doing science[1]. To see why I think this, see my LW question (including the considerations in my answer).I now want to test how intelligent orcas are as soon...

Published on December 3, 2024 5:27 PM GMTIn the spirit of the season, you can book a call with me to help w/ your interp project (no large coding though)Would you like someone to:Review your paper or code? Brainstorm ideas on next steps? How to best communicate your results? Discuss conceptual problems Obvious Advice (e.g. being affected by SAD because it's winter, not exercising,...

Published on December 3, 2024 12:30 PM GMTFor our annual update on how Balsa is doing, I am turning the floor over to Jennifer Chen, who is the only person working full time on Balsa Research. For my general overview of giving opportunities, see my post from last week. Previously: The 2023 Balsa Research update post, Repeal the Jones Act of 1920. tl;dr: In...

Published on December 3, 2024 12:20 PM GMT Our kids have been riding the bus since they were little, though less often since we started sharing a car almost four years ago. For a while the oldest, age 10, has been asking when they could start taking the bus on their own. A few days ago they did, and it went well! We worked...

Published on December 3, 2024 10:44 AM GMTTLDRLast weekend, I attended an AI evals hackathon organized by Arcadia, where the outputs were proposals for AISI’s evals bounty. My main learnings:Keep your evals simple, measuring one specific capability.It is surprisingly easy to come up with ideas, once you have seen examples.AISI extended the deadline for submissions by two weeks. Translation: they want more ideas!Story of...

Published on December 3, 2024 10:11 AM GMTTom did the original thinking; Rose helped with later thinking, structure and writing.Some plans for AI governance involve centralising western AGI development.[1] Would this actually be a good idea? We don’t think this question has been analysed in enough detail, given how important it is. In this post, we’re going to:Explore the strategic implications of having one project instead...

Published on December 3, 2024 9:29 AM GMTPrefaceSeveral friends have asked me about what psychological effects I think could affect human judgement about x-risk.  This isn't a complete answer, but in 2018 I wrote a draft of "AI Research Considerations for Human Existential Safety" (ARCHES) that included an overview of cognitive biases I thought (and still think) will impair AI risk assessments. Many cognitive...

Published on December 3, 2024 5:26 AM GMTEpistemic status: brief writeup of some interesting work I found in[1][2][3][4], among other places. Probably subtly wrong in places & if you have additions / comments to make, please do. Finite state automata (FSA) can be modeled with reactions of the form A+B→C+D..mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal;...

Published on December 3, 2024 4:33 AM GMTMIRI is a nonprofit research organization with a mission of addressing the most serious hazards posed by smarter-than-human artificial intelligence. In our general strategy update and communications strategy update earlier this year, we announced a new strategy that we’re executing on at MIRI, and several new teams we’re spinning up as a result. This post serves as...