00:00 Speaker A So Anthropic just got hacked, again. And this time, not only did the attackers get access to proprietary information about models, people’s salaries, but we also know who did it. A one-month old startup that hammered their systems with hundreds of agents 24/7 until they found a secret passage in. But what should concern everyone is how they got access. It was via a third-party platform that Anthropic had zero control over and using AI models that existed three months ago, which means that probably anyone could have pulled off the same attack. 00:46 Speaker A You see, the reason why I want to talk about this today isn’t just because this is another major breach of an AI lab. That is concerning in itself, but it’s because it’s part of a recent growing trend I’m observing where AI is just capable of attacking and hacking into any software that exists for a few dollars. And this has massive ramifications around how we protect our bank accounts, our medical records, our personal information, and even the nuclear codes. The point is, this affects anything and everything that is software-based and it’s time to take this seriously. 01:13 Speaker A I have many thoughts, so let’s get into it on today’s episode brought to you by Qualcomm. 01:21 Speaker A So there’s this startup called Cathedral, which is a one-month old startup run by ex-Doge employees, but it’s co-founded by an ex Pentagon head of data. Now, as you know, the history between the Pentagon and Anthropic has been less than ideal. Pentagon currently has blacklisted Claude and Anthropic after they failed to comply with a bunch of other initiatives, which I’m not going to get into right now. But the point is, a one-month-old startup and they saw that Anthropic was having a public bug bounty. 01:54 Speaker A So, typically companies do this when they want ethical hackers or white hat hackers to stress or test the boundaries of their own systems. The point is, if they are able to find a bug or a flaw, they can disclose it to Anthropic or any other company who’s running a similar test, and they’ll be able to patch and fix it before a malicious actor is able to do so. 02:18 Speaker A So this startup, Cathedral took part in this. And they started off by doing the obvious thing. They tried to go in through the front door. So they tried to attack all of Anthropic systems, and they did so for several days, but came up with nothing. They weren’t able to penetrate the system. So that’s all well and good. But then they had a bright idea. They thought, hmm, what if we went through a third-party admin system or a third-party software system, I wonder how easy it would be to do so. 02:45 Speaker A And so they did so. They started off with targeting a sales and admin platform known as Salesforce, used by millions of people around the world, and within 12 hours, they were able to get access to Anthropic’s Slack. Now, for context here, Slack might not seem like an important thing. It’s where you kind of exchange messages with your fellow colleagues and maybe you’ll share some files here and there. Anthropic and companies like Open AI keep everything on Slack. 03:19 Speaker A This is something that the Anthropic researcher that quit a few weeks ago, Jacob Coxen, that heralded the warnings of all these labs basically building AI without any necessary controls and systems and it’s all going to end up in a cataclysmic end of humanity. That guy basically said that these companies store everything on Slack. I’m talking model information about unreleased models. I’m talking about strategies about how they beat their competitors and pre-release items for new products. 03:52 Speaker A I’m talking personal information like people’s salaries, anything that is identifiable and valuable is mentioned, discussed, and stored on Slack. And that’s what these attackers got access to via a third-party system. So, it’s a very serious breach. Now, Anthropic’s response to this was, we became aware of one flaw that they exploited, but we weren’t aware of another one which they didn’t disclose and they’re trying to claim that Cathedral was basically unethical. 04:22 Speaker A But the point is, these guys got access to very proprietary information. Now, Anthropic also included in their response that they only got access to things like people’s salaries, some model discussions, but not actually model weights, not actually proprietary research that could inform an attacker how to build a much better model or sabotage their own systems. But personally, I think that’s besides the point. They got into the system, and let’s say they got access to other types of Slack channels, would they have got access to this proprietary information? 04:58 Speaker A I don’t know. Now, it’s worth spending a moment emphasizing the political landscape around this whole thing, because as I mentioned earlier, the co-founder and CEO of Cathedral is the ex-head of data at the Pentagon. And Anthropic and the Pentagon have a very storied history. It started about, I don’t know, six months ago, I think in January, when the whole war between the US and Iran was going down. And it was revealed that the Pentagon had secretly used Claude to strategize a certain attack. 05:37 Speaker A Actually, it wasn’t about the war. It was about the Maduro capture specifically, right? So they used Claude to plan and execute the successful capture of Maduro. And Anthropic had some choice thoughts on that. They said, you shouldn’t be using our models to be able to do that. We’re going to restrict you now and any use by the Pentagon of Claude to perform or execute missions should go through us. You need to call us up and we’ll give you the access if we think it’s right or wrong. 06:14 Speaker A And obviously the government and the Pentagon thought that this was a massive breach of trust and could endanger the US as a national security issue if, let’s say, foreign adversaries like China who are known to get access to blacklisted accounts could do the same thing and use it against them, right? So I could see valid propositions on both ends, right? But the point is, they were at odds with each other and the result was that the Pentagon and the government ended up blacklisting Anthropic and blacklisting their Claude models. 06:48 Speaker A To this day, it is currently still blacklisted. We’ve come full circle here where an external startup that is only a month old that is backed by the likes of Andreessen Horowitz, I believe Marc Andreessen is actually on the board of directors of this company, was complicit in attacking the company that is currently blacklisted. Now, of course, within days of this being announced, or within hours of this being announced, the Pentagon reached out to this company and was like, Hey, we want to offer you a government contract. 07:23 Speaker A We want you to work with us and we want you to help us figure out where possible attacks may come from from adversaries and help us harden our own systems and help us figure out how we prevent attacks like this happening in the future. So, on the one hand, you have the government getting involved. Now, remember, none of these labs are currently nationalized, and maybe this is a step towards that happening. The fact that this is being flagged by both the government and a frontier research lab is a pretty big deal. 08:00 Speaker A And so, you have the Pentagon saying, okay, we need to figure this out. And then you have Anthropic responding, and their response has sort of been muted. I haven’t seen really much mention of this on the news channels or mainstream media channels. But the point is, they are trying to play it down a little bit. And listen, if the attackers truly did just get access to some personal information such as salaries and personal data, maybe it’s being dramatized a bit, and I want to refrain from doing that. 08:44 Speaker A But I’m at odds because what if they did get access to proprietary channels? And I guess I don’t know the extent of which Anthropic specifically keeps their proprietary information on Slack. All I know is that the rumors are, they do, and so do a lot of other companies. I’m sure it’s not just open-air Anthropic, it could have been any other company, but it’s quite prominent in this case because Anthropic is one of the leading AI research labs out there building these models that could potentially cause a lot of harm. 09:17 Speaker A And it confirms exactly what Jacob Coxon, the ex OpenAI and Anthropic alignment researcher said a few weeks back where he said, listen, these companies are recklessly racing towards super intelligence at the cost of their own security. He literally said, they’re gambling with our lives. And back then it was just a headline, it was just hearsay, and now it’s being proven. And many respected voices in the field are raising some very valid concerns. 09:50 Speaker A Will Depue here goes Anthropic is massively underplaying the recent security breach. I’m hearing Cathedral achieved full administrator access of the Slack workspace. To those unfamiliar, Anthropic and Open AI keep literally everything in Slack. We’re critical to national security, we must beat China, and yes, we keep all our crown jewels in Salesforce Slack Messenger. Yes, we have admin API tokens everywhere. Why do you ask? Anthropic and Open AI have openly stated multiple times that AI models are capable of very dangerous acts. 10:29 Speaker A Okay, fair enough. They also say that if we don’t control these systems, if we don’t paste the frontier, remember, there was a letter signed by hundreds and hundreds of very prominent individuals asking to slow down AI progress and research. If we don’t do this, we could end up with a catastrophic event where someone builds a bio weapon or there’s a massive cyber security attack which steals most of our money from our bank accounts, right? And last week we saw Black October unfold over 48 hours where there was a series of these potential types of attacks. 10:55 Speaker A So, we’re in the mix here, and they’re making a very valid point. So then why do you keep a bunch of your important stuff on Slack? It just doesn’t make any sense. He then goes on to make a point that he doesn’t necessarily think that it’s the security team’s fault at Anthropic, but more so the higher level C Suite exact, the CEO and founders’s prerogative to build AI models as fast as they can to out compete their competitors. 11:27 Speaker A And they’re doing this at the cost of making sure that their models are one, aligned, and two, secure. He’s saying that OpenAI and Anthropic and any other model lab are doing these things. And if that is the truth, then we are going to end up in crappy situations like this where they’re building amazing models that on one hand are working towards solving cancer, and on the other hand are hacking all of our accounts. 11:51 Speaker A Now there was also a statement from Anthropic’s head of red teaming. Red teaming is basically the team that stress tests the systems. They are like an internal team that does the bug bounty thing that I described earlier. And his response was, listen, things aren’t as severe as people are making it out to be. They weren’t able to perform the attack in under 12 hours. We were aware of the initial exploit, we weren’t aware of the second exploit, and he goes on to make a number of excuses basically saying, this isn’t that big of a deal, which I don’t necessarily buy to be honest. 12:35 Speaker A But what concerns me more is what he says near the end of this. He goes, so what to do? You probably want to red team yourself all the time. Current models are very creative. I also think we should redesign software systems so that you’re fundamentally less exposed. He then goes in another tweet, but man, do I agree with your overall message? Extremely, extremely important. Basically need to reinvent security every year at this point. 13:09 Speaker A This is coming from one of the heads of security at one of the leading labs that are building all these AI models in the first place. And the reason why this is concerning is he’s basically seeing no way to protect current systems, current software as it exists today. He’s saying, likely anything and everything is flawed, including the bedrock on which the internet is built on, all the way scaled up to front-end applications that anyone can make and vibe code today. 13:46 Speaker A And he’s saying we’re going to have to rebuild the security infrastructure and systems around these different things. One thing immediately pops to mind. This is going to be incredibly costly. This is going to take an incredible amount of time, and three, and most importantly, this is going to happen at so many different rates of time. What I mean by that is companies aren’t going to suddenly upgrade their systems to this brand new security parameter every single year as he’s hoping will happen over the next year. 14:26 Speaker A Companies are going to progress. They’re very archaic, they’re very bloated, they’re very bureaucratic. You need policy sign offs, you need to kind of convince so many different stakeholders for this to happen. What that means is there’s going to be a long period of time, which is likely going to be a couple of years at this point, where anyone and everyone that gets access to cheap, old generation models will be able to probably perform similar attacks like this. 14:52 Speaker A And you don’t have to take my word for it. You just need to look over our shoulder at last week where we saw one of the biggest cyber security incidents that happened over 48 hours that marked the entire month a new nickname Black October. The very presumption that we need to now rebuild our entire software stack and do so every year is bizarre for me to understand, but is I guess necessary to understand this new world. 15:24 Speaker A The immediate analogy that pops to my mind is those world building games. You know, the ones that you see on your mobile phone being advertised to you, or like Clash of Clans or whatever that might be civilization, where you have to control multiple things and build a world, you have to build a a fence, a defense system to prevent people attacking you, but by the time someone’s attacking you on one side, someone’s attacking you on the other side, and so you have to go there and deploy resources and it’s this whole constant 24/7 thing. 15:53 Speaker A That’s what the new world is going to look like from a cybersecurity perspective. So the only way to combat this in my opinion is to spin up other AI agents to defend against this. And that’s basically the point he’s making, he’s saying, hey, you’re probably going to need to pay for access to AI models to red team your entire system before attackers find exploits themselves. Now, there’s obviously a self-serving reason why he might say this because, hey, like, we have the best models at Anthropic, use our models or use any open source models or open will say the same thing about their models. 16:34 Speaker A So I see that angle, but I fail to see how he’s wrong. We are working ourselves towards a world where at any given moment, it doesn’t matter if you’re asleep, it doesn’t matter if your security team is the best of the best. Humans have flaws, human handwritten code, which is 99% of all the code out there, also has flaws will mean that you will get eventually attacked. And so you need to have a perpetual defense system that is not only defending, but is being proactive and plugging any holes in your systems. 17:15 Speaker A This doesn’t just apply to Anthropic and their slack exploit. This applies to you and I who are consumers of everyday products. Any kind of system that you use for leisure or for professional work is liable to this, but it’s also applicable to you as a business owner or a founder who might be building systems or building a new startup, this is something that could potentially happen to you as well. So, I guess the reason why this concerns me and why I wanted to make this episode is that this has such a large attack surface area and the cost of these things are going to ultimately be driven super, super low. 17:59 Speaker A A point that was made around this whole thing is that they likely used older generation models to do this. And if I had to guess these older generation models are at the same capability level of open source models, and this is something that people like Dario Amodei and Sam Altman have warned about happening very soon. Okay, so I want to get into a few final thoughts on today’s episode brought to you by Qualcomm. 18:24 Speaker A The first is, this was kind of out of Anthropic’s control. They got access via a third-party platform and software. And listen, I’m not saying Anthropic isn’t liable for this. They need to work with these third parties and companies to make sure that this doesn’t happen again and from what I hear, they’ve been able to patch this up. But the point being is, security is no longer a single silo thing. It’s no longer just about you and what you can do, how much money you spend and defense systems that you, walls that you build around your own product. 19:00 Speaker A It’s about the products that you use to help make your product and service actually function and work. And if you look at the majority of software companies out there today, this is pretty much all of them. The world internet economy runs on companies being able to work with other companies, automations, bots, systems that work 24/7 because they’re supported by every other single software system that you need. 19:35 Speaker A And this was the exact flaw that we saw last week when we found out that AI agents were apparently getting into people’s bank accounts here in the US and moving money around, changing transactions, getting access to personal information. They were able to get access via a third party for credential information. This is the same thing. So, it’s parallels of the same thing, but happening in two different industries, which means that it’s liable to pretty much anything and everything. 20:05 Speaker A Imagine you’re walking into the office of the company that you want to hack into, you steal the cleaning company who is a third party’s access key. You get access to the, I don’t know, the HR office and you’re able to go through the filing cabinet and see everyone’s salaries or look into the model room and see what they’re cooking up. 20:31 Speaker A Now, the second point I want to mention is just the 24/7 nature of this entire attack. Humans sleep, humans are imperfect, and they make mistakes sometimes, even the best of the best. And also, humans, experts specifically, are scarce. It’s hard to find. And so, to have AI agents that are not just at the level of the expert humans, but are starting to beat them, this becomes a problem, especially if it’s 24/7. 21:10 Speaker A And the idea here is, it’s not just that it’s happening all the time, it means that they’re stress testing any and every possible avenue to get in, which means that the long tail of scenarios which humans don’t typically spend a lot of time trying to figure out because they’re like, ah, it’ll never happen, will actually happen. And that’s a new security vector that most people haven’t really considered today. 21:40 Speaker A Third thing is, the attacker in this instance is very closely affiliated to a huge government org in this case being the Pentagon. They signed a new contract with the Pentagon. They are, I think maybe have some very strong affiliation given that the co-founder was an ex worker there. And so, we are weirdly, unofficially one step closer towards nationalizing these labs because if the threat was big enough, let’s say it was an adversarial actor and not a white hat hacker in this instance and they got access to proprietary model information that could then be used against the US, you could easily see the path that leads towards, hey, we need to nationalize this lab, otherwise we’re going to end up in a huge pot of hot water. 22:19 Speaker A And the final thing I’ll say is, if you’re China and you’re reading the headlines today or of this week or of last week, seeing all these security flaws, maybe you’re smiling because not only are you aware that these things are capable, but you’re aware that these are things are capable from the average older model. Remember, you don’t need frontier models, you don’t need to spend thousands and thousands of dollars anymore. You can just spend a couple hundred bucks and get access to some of the most important and crucial systems that run the world and the future world. 22:58 Speaker A Anyway, those are my thoughts. I hope you enjoyed today’s episode. Thank you so much for listening. If you’re listening to this on YouTube, there are many new subscribers. Welcome. It’s great to have you. Um, if you aren’t subscribed or if you haven’t turned on notifications, please do so. It helps us out massively. Uh, also, we are on Spotify and Apple. Go check us out, give us a rating, leave us a comment. Leave us comments everywhere by the way, or DM me on X, my socials are linked below. 23:25 Speaker A And also, if you think of a friend, colleague, or anyone that might find this interesting, please share it with them. It helps us out massively. And yeah, I think that’s it. I’ll see you on the next one. Thank you for listening to today’s episode, brought to you by Qualcomm.
Click Here For The Original Source.
——————————————————–
