Bookmarks (696)

Published on July 26, 2024 4:32 PM GMTThe Bay Area rationalist community has an entry problem! Lots of listed groups are dead, the last centralized index disappeared, communication moved to private discord and slacks. This is bad, so we're making a new index, hopefully up to date and as complete as we can!CommunicationDiscord: Bay Area Rationalists: https://discord.gg/EpG4xUVKtfEmail Group: BayAreaLessWrong: https://groups.google.com/g/bayarealesswrongLocal Meetup GroupsTaco Tuesday: by...

Published on July 26, 2024 2:08 PM GMTOn 75 percent or more of the residential land in most major American cities it is illegal to build anything other than a detached single-family home. 95.8 percent of total residential land area in California is zoned as single-family-only, which is 30 percent of all land in the state. Restrictive zoning regulations such as these probably lower...

Published on July 26, 2024 11:18 AM GMT“You should practise acceptance.”“What do you mean?”“You’re being too hard on yourself, you should accept yourself more.”“Sounds reasonable. I think I get the “vibe” of acceptance, but I still feel confused. How do I practise acceptance?”“Mmm I haven’t thought hard about this before. I’m guessing it’s about loving every part of you, even your weaknesses and failures....

Published on July 26, 2024 7:23 AM GMT(Crossposted from Twitter)I'm skeptical that Universal Basic Income can get rid of grinding poverty, since somehow humanity's 100-fold productivity increase (since the days of agriculture) didn't eliminate poverty.Some of my friends reply, "What do you mean, poverty is still around?  'Poor' people today, in Western countries, have a lot to legitimately be miserable about, don't get me...

Published on July 26, 2024 12:33 AM GMTA Solomonoff inductor walks into a bar in a foreign land. (Stop me if you’ve heard this one before.) The bartender, who is also a Solomonoff inductor, asks “What’ll it be?”. The customer looks around at what the other patrons are having, points to an unfamiliar drink, and says “One of those, please.”. The bartender points to...

Published on July 25, 2024 10:03 PM GMTStatus: Thought experiment for fun Imagine a world in which the gambler's fallacy is fundamentally true. Functionally, lets suppose there's a magical force that tracks a thinking being's expectation of any particular outcome, and then mysteriously increases the likelihood of said outcome the more often it had physically plausible opportunity to occur and did not[1]. (This resets upon...

Published on July 25, 2024 10:00 PM GMTWork done at FAR AI.There has been a lot of conceptual work on mesa-optimizers: neural networks that develop internal goals that may differ from their training objectives (the inner alignment problem). There is an abundance of good ideas for empirical work (find search in a NN, interpret it), but very little actual .mjx-chtml {display: inline-block; line-height: 0; text-indent: 0;...

Published on July 25, 2024 8:55 PM GMTAdversarial vulnerabilities have long been an issue in various ML systems. Large language models (LLMs) are no exception, suffering from issues such as jailbreaks: adversarial prompts that bypass model safeguards. At the same time, scale has led to remarkable advances in the capabilities of LLMs, leading us to ask: to what extent can scale help solve robustness?...

Published on July 25, 2024 7:11 PM GMT2024-07-25: I'm organising a little reading group, undecided between in-person in London or remote. My own questions I want to get the answers to are in https://www.patrickstevens.co.uk/posts/2024-07-25-lob-theorem/ . Writing this here so we have a place to start coordinating how we do this. I'm away on holiday until next Sunday so won't be doing anything about the...

Published on July 25, 2024 2:58 PM GMTThis is based on our recent preprint paper “Constructing Benchmarks and Interventions for Combating Hallucinations in LLMs” by Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov.Code - https://github.com/technion-cs-nlp/hallucination-mitigationSummaryIn a question-answering (QA) setup, hallucinations can occur when a model generates incorrect answers.Three types of hallucinations can be identified based on the model's knowledge of the correct (golden)...

Published on July 25, 2024 3:58 PM GMTGoogle DeepMind reports on a system for solving mathematical problems that allegedly is able to give complete solutions to four of the six problems on the 2024 IMO, putting it near the top of the silver-medal category.Well, actually, two systems for solving mathematical problems: AlphaProof, which is more general-purpose, and AlphaGeometry, which is specifically for geometry problems....

Published on July 25, 2024 3:54 PM GMTThere doesn't seem to be an arxiv PDF out yet for AlphaProof, so I'm linking the announcement post. Relevant quote: AlphaProof is a system that trains itself to prove mathematical statements in the formal language Lean. It couples a pre-trained language model with the AlphaZero reinforcement learning algorithm, which previously taught itself how to master the games...

Published on July 25, 2024 3:49 PM GMTThis is an edited transcription of the final presentation I gave for the AI safety camp cohort of early 2024. It describes some of what the project is aiming for, and some motivation. Here's a link to the slides. See this post for a more detailed and technical overview of the problem.This is the presentation for the...

Published on July 25, 2024 1:50 PM GMTWe got two big model releases this week. GPT-4o Mini is covered here. Llama 3.1-405B (and 70B and 8B) is mostly covered in yesterday’s post, this has some follow up. Table of Contents Introduction. Table of Contents. Language Models Offer Mundane Utility. All your coding are belong to us. Language Models Don’t Offer Mundane Utility. Math is...

Published on July 25, 2024 11:18 AM GMTSammy Martin, Polaris VenturesAs AI systems become more integrated into society, we face potential societal-scale risks that current regulations fail to address. These risks include cooperation failures, structural failures from opaque decision-making, and AI-enabled totalitarian control. We propose enhancing LLM-based AI Constitutions and Model Specifications to mitigate these risks by implementing specific behaviours aimed at improving AI...

Published on July 25, 2024 1:12 AM GMTProduced as part of the ML Alignment & Theory Scholars Program - Summer 2024 CohortThis post presents the basic story behind my current research direction at MATS. I'm looking for feedback on whether the basic premises hold up, and which research questions should I prioritize.The role of investors in fueling the AGI race is unclearMost Frontier AI Labs...

Published on July 24, 2024 10:41 PM GMTThis post lays out a framework I’m currently using for thinking about when AI systems will seek power in problematic ways. I think this framework adds useful structure to the too-often-left-amorphous “instrumental convergence thesis,” and that it helps us recast the classic argument for existential risk from misaligned AI in a revealing way. In particular, I suggest,...

Published on July 24, 2024 7:40 PM GMTIt’s here. The horse has left the barn. Llama-3.1-405B, and also Llama-3.1-70B and Llama-3.1-8B, have been released, and are now open weights. Early indications are that these are very good models. They were likely the best open weight models of their respective sizes at time of release. Zuckerberg claims that open weights models are now competitive with...

Published on July 24, 2024 6:53 PM GMTExtensive collection of memes compiled by Victor Li and other contributors on AI Safety Info, mostly using memes by AI Notkilleveryoneism Memes.Memes can, at their best, convey key points in a sticky and easily sharable form. Having a index seems potentially quite helpful, and also is a fun resource, so we've adopted it along with other living...

Published on July 23, 2024 4:07 PM GMTThis is an informal research note. It is the result of a few-day exploration into RMU through the lens of model internals. Code to reproduce the main result is available here.This work was produced as part of Ethan Perez's stream in the ML Alignment & Theory Scholars Program - Summer 2024 Cohort. Thanks to Nina Panickssery, Mrinank Sharma,...

Published on July 23, 2024 12:50 PM GMTIt is monthly roundup time. I invite readers who want to hang out and get lunch in NYC later this week to come on Thursday at Bhatti Indian Grill (27th and Lexington) at noon. I plan to cover the UBI study in its own post soon. I cover Nate Silver’s evisceration of the 538 presidential election model,...

Published on July 23, 2024 12:43 PM GMTEpistemic status: possibly trivial, but I hadn't heard it before.TL;DR: What I thought of as a "flaw" in PCA—its inability to isolate pure metrics—might actually be a feature that aligns with our cognitive processes. We often think in terms of composite concepts (e.g., "Age + correlated attributes") rather than pure metrics, and this composite thinking might be...

Published on July 22, 2024 9:16 PM GMTCan a sin tax solve the ransomware problem? The following is a proposal for a gradual increase in the taxation of ransom payments:The ransom demanded by an attacker is a function of the expected likelihood that the victim will pay that amount. If a high tax is legally enforced, the market will adapt: "optimal" ransoms will account for...

Published on July 23, 2024 3:27 AM GMTThings I'm fairly confident in: I should take colds in general more seriously than I did pre-pandemic: Staying at home with cold symptoms is good. General masking during cold season is good. We should have air filters in all public indoor spaces. Long covid is real and we should keep getting covid shots and avoid infections. Things...

Published on July 23, 2024 3:03 AM GMTMy rationalist group house near Boston has a room opening (details here), and I'm wondering if there's a Schelling point for listing that sort of thing. I posted the opening here and to the relevant local Facebook groups, but I'm wondering where might be good places to post it that people in the community who are moving...

Published on July 23, 2024 2:55 AM GMTWe have a room opening up in a rationalist house near Central Square, Cambridge (Boston). It is a 4-bedroom, 3-bathroom apartment with two floors, with a large kitchen and living room plus additional common space upstairs. There’s a washer, dryer, and dishwasher in-unit. It’s an 8-minute walk from Central station and a 15-minute walk from Kendall station....

Published on July 23, 2024 2:00 AM GMTThere have been a lot of D&D.Sci scenarios, but there's a lot of variance between them in complexity and quality.  Some are more difficult, and might not be a good place to start, while others are much simpler - some were very good, while others on reflection didn't flow quite right.  Unfortunately, LW karma doesn't track the...

Published on July 23, 2024 1:45 AM GMTThis is my advice for careers in empirical ML research that might help AI safety (ML Safety). Other ways to improve AI safety, such as through AI governance and strategy, might be more impactful than ML safety research (I generally think they are). Skills can be complementary, so this advice might also help AI governance professionals build...

Published on July 22, 2024 8:20 PM GMTAt 2024's Manifest, Robin Hanson gave a talk (in his usual sweeping polymathic style) on cultural drift - a phenomenon for which he thinks "there's going to be hell to pay", and about which he is "scared, because this is a really big fundamental problem".Watch the recording of the 30:00 minute talk here (followed by Q&A).His argument...

Published on July 21, 2024 6:18 PM GMTRecently, there has been discussion on how to make model weight exfiltration more difficult, such as Ryan Greenblatt's proposal for upload limits. A key factor enabling this is that the critical data we want to protect (model weights) are very large files. Current models can have trillions of parameters, which translates to terabytes of data. Ryan calculated...

Published on July 21, 2024 2:15 PM GMTI was listening to Anders Sandberg talk about "humble futures" (i.e., futures that may be considered good in a sober non-"let's tile the universe with X" way), and started wondering whether training (not yet proven safe) AIs to have such "humble" scope-insensitive-ish goals -- which seems more tractable than (complete) value alignment -- might disincentivize the AI...

Published on July 21, 2024 1:24 PM GMTConsider an entire function (complex-differentiable everywhere) f(z).mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin:...

Published on July 21, 2024 12:37 PM GMTThis idea is so simple that I'm sure it's been had by someone somewhere. Suppose we have some method to make really smart honest AIs that do not have goals. Let's say it's a yes/no oracle. Our aimless ace. But we want to accomplish stuff! AIcorp wants the printmoneynow.py. I'm worried that if I train the ace...

Published on July 21, 2024 12:15 PM GMTThe prevailing notion in AI safety circles is that a pivotal act—an action that decisively alters the trajectory of artificial intelligence development—requires superhuman AGI, which itself poses extreme risks. I challenge this assumption.Consider a pivotal act like "disable all GPUs globally." This could potentially be achieved through less advanced means, such as a sophisticated computer virus akin...

Published on July 20, 2024 3:45 PM GMTHeads up: This is not really a post about rationality. That said, there are few times in your life when you are more prone to biases and emotional overrides than when you engage in dating. We are emotional creatures above all. Knowing principles of rational thinking and tactics for overcoming biased thinking will not protect you from emotional...

Published on July 21, 2024 2:20 AM GMT With the Red Line shut down north of Kendall for repairs I've been walking to the Green Line at Ball Sq. It's definitely a slower commute, but it's not bad. Except for 150 painful feet. Here's how I walk to the station (240ft): And here's what I want to do (90ft): They've put up enough of...

Published on July 20, 2024 9:43 PM GMTI don't work in cyber security, so others will have to teach me. I'm interested in the question of how AI systems can become private. How to make communications with an AI system as protected as the confessional. Some AI capabilities are throttled not for public interest reasons but because if those private conversations became public, the...

Published on July 20, 2024 3:08 PM GMTHenry George’s 1879 book Progress & Poverty was the second best-selling book in the entire world during the 1880s and 1890s, outsold by only the Bible. Nobody knows exactly how many copies it sold during those two decades since nobody was keeping track, but it definitely sold at least several millions of copies for sure. The Progressive...

Published on July 20, 2024 5:03 AM GMTThis post builds on my previous post making the case that squiggle-maximizers are plausible. The argument I presented was a deliberately simplified one, though, and glossed over several possible issues. In this post I'll raise and explore three broad objections. (Before looking at mine, I encourage you to think of your own biggest objections to the argument,...

Published on July 19, 2024 2:07 PM GMTA short summary of the paper is presented below.TL;DR: We develop a robust method to detect when an LLM is lying based on the internal model activations, making the following contributions: (i) We demonstrate the existence of a two-dimensional subspace, along which the activation vectors of true and false statements can be separated. Notably, this finding is...

Published on July 19, 2024 1:59 PM GMTHiroshi Yamakawa1,2,3,41 The University of Tokyo, Tokyo, Japan2 AI Alignment Network, Tokyo, Japan3 The Whole Brain Architecture Initiative, Tokyo, Japan4 RIKEN, Tokyo, JapanEven in a society composed of digital life forms (DLFs) with advanced autonomy, there is no guarantee that the risks of extinction from environmental destruction and hostile interactions through powerful technologies can be avoided. Through thought-process diagrams, this study...

Published on July 19, 2024 4:10 PM GMTNew paper from the Google DeepMind mechanistic interpretability team, led by Sen Rajamanoharan!We introduce JumpReLU SAEs, a new SAE architecture that replaces the standard ReLUs with discontinuous JumpReLU activations, and seems to be (narrowly) state of the art over existing methods like TopK and Gated SAEs for achieving high reconstruction at a given sparsity level, without a...

Published on July 19, 2024 1:03 PM GMTWhatever each culture grows and manufactures cannot fail to be here at all times and in great profusion. Here merchant vessels arrive carrying these many commodities from every region in every season and even at every equinox, so that the city takes on the appearance of a sort of common market for the world.Your farmlands are Egypt,...

Published on July 19, 2024 12:23 PM GMTThe BlueDot Impact write-up for scalable oversight seems to suggest that people have given up on iterated distillation and amplification (IDA) working. I haven’t really seen much research here, but is that actually the case?I know that Ought is now pretty much inactive because their attempts at factored cognition failed and this has made many people pessimistic...

Published on July 19, 2024 12:31 AM GMTAI Alignment is my motivating context but this could apply elsewhere too.The nascent field of AI Alignment research is pretty happening these days. There are multiple orgs and dozens to low hundreds of full-time researchers pursuing approaches to ensure AI goes well for humanity. Many are heartened that there's at least some good research happening, at least...

Published on July 18, 2024 10:18 PM GMTAdam Mastroianni writes about "actually caring about stuff, and for the right reasons", rather than just LARPing. The opening is excerpted below.I once saw someone give a talk about a tiny intervention that caused a gigantic effect, something like, “We gave high school seniors a hearty slap on the back and then they scored 500 points higher...

Published on July 18, 2024 7:02 PM GMTThe current cohort of the ML Alignment & Theory Scholars Program, MATS 6.0, had a unique application process and its broadest selection of mentors yet, with 40 mentors to apply to. I was invited to interview with twelve mentors and was accepted by five (which I later learned was an unusual number of interviews and offers in...

Published on July 18, 2024 6:44 PM GMTA few months ago, Rob Bensinger made a rather long post (that even got curated) in which he expressed his views on several questions related to personal identity and anticipated experiences in the context of potential uploading and emulation. A critical implicit assumption behind the exposition and reasoning he offered was the adoption of what I have...

Published on July 18, 2024 6:37 PM GMTI know little about Rob Henderson except that he wrote a well-received memoir and that he really really really wants you to remember that he invented the concept of “luxury beliefs”. In his own words, these are:ideas and opinions that confer status on the upper class at very little cost, while often inflicting costs on the lower...

Published on July 17, 2024 6:26 PM GMTSummaryAgents might fail to peacefully trade in high-stakes negotiations. Such bargaining failures can have catastrophic consequences, including great power conflicts, and AI flash wars. This post is a distillation of DiGiovanni et al. (2024) (DCM), whose central result is that agents that are sufficiently transparent to each other have individual incentives to avoid catastrophic bargaining failures. More precisely, DCM constructs strategies...