ML for SWEs #22: 1000+ AI Agents Make a Civilization, xAI Creates the Largest Supercomputer in 4 Months, Possible AI Research Fraud, and More

Machine learning resources and updates 2024-09-09

You may notice an update to the format this week. I’m going to take more time to write about why something is important and highlight the most important developments. I’m also going to include important discussions that occurred throughout the week. I’ve also gotten rid of the table of contents now that Substack adds it automatically. Of course, the full reading list will still be available to supporters. 

Finally, I’m going to add some information about the job market and the skills to succeed in ML. I’m still figuring out the best way to do that, but for now I’ve added a job skills section with resources for how to learn the skills to my ML road map. Let me know what you think of the new format! I also apologize for any grammatical or spelling mistakes this week. I’m excited about this format and wanted to get it together ASAP even though my sources weren’t as organized as I’d like them to be.

Here are the most important machine learning resources and updates from the past week. Follow me on X and/or LinkedIn for more frequent posts and updates. You can find last week's updates here.

Support the Society's Backend community for just $1/mo to get my full reading list each week. Society's Backend is reader-supported. Thanks to all paying subscribers! 😊

Highlights

Anthropic’s Customer Support RAG

Anthropic partnered with Pietro Schirano to create a customer support agent using RAG. The entire repo is available on GitHub for anyone to use. This is an excellent starting point for any RAG system and has been added to the Claude Quickstart repo. I’d encourage anyone looking to build something like to give it a try. It’s reported to be easily scalable and developer friendly. I know many of you build AI systems so I figured this would be valuable. I’m debating using it to organize all these sources I come up with and making them easily searchable. Let me know if you think that would be useful.

Project Sid

A Minecraft server was developed with 1000+ autonomous AI agents to create a virtual world and the results were astounding. I’m going to link to the post on X with a video detailing the process of creating this world and its outcome. It shows the capabilities of AI agents working together and how they can interact at an unprecedented scale. The agents created religion, government, culture, and economy and more.

Understanding Microsoft's “Aurora: A Foundation Model of the Atmosphere” [Breakdowns]

Microsoft introduced Aurora, a 1.3 billion parameter foundation model designed for high-resolution weather and atmospheric forecasting. Aurora leverages a 3D Perceiver encoder and Swin Transformer U-Net backbone to handle heterogeneous data and predict weather patterns with high accuracy, even with limited training data. It outperforms traditional simulation tools and specialized deep learning models by producing 5-day global air pollution predictions and 10-day high-resolution weather forecasts quickly. Devansh wrote an excellent breakdown on the Aurora paper here.

xAI’s massive training cluster

xAI has launched the Colossus supercomputer, currently the world's most powerful AI training system, featuring 100,000 Nvidia H100 GPUs. Plans are in place to double its capacity to 200,000 GPUs, combining 50,000 Nvidia H100 and H200 models, in the coming months. This is an incredible feat because it took only about 4 months to get it set up. This is a logistic masterpiece and great collaboration between Dell, Nvidia, and xAI.

🥇Top ML Papers of the Week

I always highlight paper overviews and this week has had many interesting papers released. Key papers include AlphaProteo, which significantly improves protein binding affinities, and MemLong, which extends LLM context length up to 80k tokens. Additionally, research shows generative AI tools boost software developers' productivity by over 26%. Check out the summaries here.

Discussions

AlphaProteo

Google released AlphaProteo, an AI model that designs novel proteins for drug development. There was some interesting discussion on X that I had to get involved in where the sentiment toward the AI Google puts out to help with drug development is negative. The general sentiment seems to be that Google is putting out these models and they aren’t helping with anything. There was even a large account that stated Google should be focusing on curing cancer and diabetes instead of these models as if those two tasks are simple.

I had to clarify that misinformation and explain how beneficial these models have been. It resulted in me getting blocked, but one of the reasons I’m on X is to help shake misinformation. For a simple explanation, Google’s AI models help design novel proteins that can be used to treat illnesses and symptoms. The AI has taken a process from hours to minutes. This is only a small portion of drug development. After these proteins are determined, there are years of other processes before a drug hits the market. Drug development is a complex and time-consuming process and most people don’t seem to understand this. If you want a more comprehensive timeline for drug development, here’s a good explanation.

OpenAI’s Large Price Tag

There’s been a lot of discussion surrounding an estimate for OpenAI’s pricing going forward. The estimate (unconfirmed as far as I can tell) puts OpenAI’s cost for monthly subscription at ~$2000. This is enterprise pricing. Many people on X are making posts as if this will be charged to consumers. This is a pretty fair enterprise price in my opinion as it can bring in this much value and enterprise-grade support must also be provided.

TIME’s Top 100 Influential People in AI

TIME posted their top 100 most influential people in AI. They included AI influencers and celebrities while leaving out many people who have contributed significantly to actual technical contributions. A glaring omission is Elon Musk who has, love him or hate, contributed significantly by facilitating a lot of cutting-edge AI research, development, and products. This has sparked discussion about how truthful journalism really is.

Reflection 70B’s Misfire

I’m not a huge fan of highlighting when people/companies make mistakes unless it’s important and I feel this is. A company released a 70B parameter model claiming it outperformed much larger models. Upon independent inspection, the claims couldn’t be verified and there was evidence shown that release was entirely truthful. For a more detailed report, check out this post on X. It’s a bit more negative than I would like, but it does show all the claims found across X over the past few days. Here’s another post about the current state of analyzing it. If something comes up where Reflection 70B is actually really good, I’ll let you know.

CosmDallas is a Massive Dome to Create an Immersive VR-like Experience

I’m including this one just because I think it’s cool and it also solves a lot of the problems with current VR. There is a dome in Dallas that displays football games from a camera inside the actual stadium. The dome allows for many fans to fit inside and experience the game via the camera. Watching sports is very much a group experience and this method of doing so captures the benefits of VR while creating a more immersive experience.