hckrnws
I've been working on that for a client since yesterday (as a fractional CTO). Pretty hectic, basically nothing really works and we don't know yet if all data is lost or if anything is recoverable or when AWS UAE will become functional again so we can recover that region.
Finally, I have a very good argument for multi-region deployments ;))
that's my go to website atm: https://health.aws.amazon.com/health/status
We didn't do multi-region deployments, but we did store database backups in a separate region just in case something really bad happened and our AWS region became unavailable. Also had a plan/some ready Terraform stuff in order to start setting up a deployment if it became apparent that the region wasn't coming back anytime soon.
IMO, if you're using AWS and not replicating your data somewhere else, this should be an eye-opener for you.
Not sure why everyone read this as me doing anything here, I'm a fractional CTO, which is kind of an advisor. Nothing invaluable will be lost tho. It's not the core platform, just a localized version for specific customers in the region.
Severity: Disrupted
So if data won't be recoverable you all will mark it something like "Status: FUBAR" or some equivalent term?
I wonder if ‘Apocalypse’ or ‘Molten Slag’ would be considered professional enough.
I'd accept "X_X" as status.
Status: Torpedoed
FUBAR, I would vote for that
What do you mean 'finally' - surely 'redundancy' or 'natural disaster' is reason enough.
To a lot of managers/startup execs this is something like "we won't ever need this". And I'd agree to some extent for not so important/rebuildable services. Depends on what you need. In startups, you don't have infinite time you can spend on stuff. But this makes a good case: if a geographical region only has one AWS region, don't keep data or run services that can't be easily rebuild somewhere else. In europe you can just pick two AWS regions and you stay in the same regions, UAE not so much.
There’s a reason literally every enterprise of any size in the early 00s not only had a DR site, but a DR plan they would test multiple times per year.
1. Just having a copy in two places doesn’t mean the copies are both good or that you can get DR online in any reasonable amount of time.
2. You quickly figure out all the things you thought “weren’t important” that prevent you from actually doing a successful DR test.
They amount of things that “cloud first” people just assume naturally takes care of itself because it’s in the cloud always amuses me.
You should have that for your main data, yes. I insist on that as well. Making backups, cold storage, and getting them back online. Yearly is not enough. My point was that not everything needs a backup or a DR. Spend your time wisely, we're talking startups, not enterprise. Can't talk more specifics obviously
The fact half the internet was down the last time US east was down tells me it’s not just startups that foolishly think they don’t need actively tested dr copies and plans when they move to the cloud.
I don't think this changes anything. It is always the same argument for me.
"How often do those happen?"
That was my point: now we have a very specific case we can argue. I always used the "what if a plane crashes into the data center" argument. Now I can say: in one of my engagements, we lost a datacenter completely because of a drone attack. That's a first for AWS as well, but we can draw from reality, not hypotheticals.
So AWS's availability zone architecture hasn't helped in this case?
If a whole region is bombed then you wouldn’t expect that region to stay up.
All AZs were affected
You don't already have a DR plan in place?
Of course, no worries. Nothing irreplaceable will be lost ;) It was meant as a general example to be used in future arguments for everybody in tech leadership.
Any backups?
> Any backups?
Yup something has to be said too about good old offline backups on fat tapes.
More recent news: https://www.businessinsider.com/amazon-data-centers-middle-e...
> Two facilities in the United Arab Emirates sustained direct hits, while a third facility in Bahrain was damaged by a drone strike "in close proximity,"
Also to add context: AWS has contracts with the US military: "The Joint Warfighting Cloud Capability (JWCC) contract enables AWS to continue providing Department of Defense (DoD) customers with secure, reliable, and mission-critical cloud services." https://aws.amazon.com/federal/defense/jwcc/ Making them a target for retaliation ofc.
Amazon is an extremely visible American company, hitting their assets carries a symbolic meaning even if the DoD wouldn't have anything running on that datacentre at all. Iran's trying to transmit a message of "we can destroy your stuff too", trying to impact the general US feeling of invulnerability.
I don't think it'll work, but they might as well try I guess.
> trying to impact the general US feeling of invulnerability
Or, perhaps, trying to defend themselves? They are being attacked, after all.
It's both, this particular counter-attack is aimed at morale rather than specifically a base launching sorties against them.
Ultimately, this war ends when America loses the political will to continue, so morale is a strategic objective for them.
America is so unused to being attacked (counter-attacked) that this needs to be explained apparently.
That's why they've been hitting residential buildings and hotels as well? They assume that because their proxies (Hamas, Hezbollah) hide in civilian structures, so does the US army?
All these attempts to justify Iranian terror demonstrate just how deep Qatari influence online has been. And even Qatar is being attacked by Iran now.
Is Hezbollah hiding in the elementary school that got bombed? Perhaps that’s where the Iranian nuclear research was done?
We attacked them. Full stop. And as far as I can tell we haven’t given them any conditions for when we will stop bombing them. In what moral framework do you have to just accept another sovereign, a vastly more powerful one, invading your country without fighting back?
It's not quite as clear-cut as it might first seem.
The school was either within or bordering a military barracks, depending which way you wish to spin it.
Al-Jazeera's article has obvious bias, but plenty of pictures and diagrams:
https://www.aljazeera.com/news/2026/3/3/questions-over-minab...
Thank you for the link. AlJazeera is blocked in my country, but I was able to read it. Why is the school called Shajareh Tayyebeh, Arabic for Good Tree? Don't the Iranians speak Persian? Why an Arabic name?
I can't guess what the USA wants other than a distraction from the raping-of- children saga, but I bet Israel would settle for "we acknowledge your right to exist and won't fund or encourage organisations that plan to harm you."
If Saudi Arabia can get there…
The world has seen what Israel does when they’re attacked. They don’t get to set moral frameworks anymore.
Agreed. But that wasn't the OP's question.
Who in your opinion sets the moral framework for defending oneself against an enemy which has sworn genocide and proven capable of destroying entire peaceful villages along their borders?
you're describing Israel here, right?
Which village in Gaza do you consider to have been peaceful?
There are no peaceful villages in Israel, they exist only due to recent violence.
You do not consider Kibbutz Beeri to have been a peaceful village bordering Gaza? Or you are just unfamiliar with the area?
I challenge you to research Kibbutz Beeri and find a non-peaceful description of the village.
It's really hard to describe "11 points in the Negev" as peaceful.
Kibbutz Beeri was created for the sole purpose of justifying a hostile land grab by a bunch of well-funded European immigrants.
[flagged]
The IDF did not starve civilians - that lie has already been disproven. I know that you'd love to repeat it until history records it, but by no objective measure was there famine nor starvation in Gaza. Other than the starvation of Hamas' hostages.
The images of "starving children" were images of children with other medical conditions. The UN reports used a metric that considered starvation at HALF the threshold used in every other conflict zone, and even with that metric only found "evidence" in a single location once.
… but they did bomb apartment buildings in Tehran. I saw that with my own eyes.
Stick to the challenge: what won't you accept? Nothing?
I accept the bombing of buildings which house those who have declared "Death to America, Death to Israel", and then have proceeded to bomb apartment buildings in Beit Shemesh.
That's a war crime.
No, that's just war.
From The United States Uniform Code of Military Justice: "Attacking civilians is unequivocally a war crime".
https://ucmj.us/is-attacking-civilians-a-war-crime/
@dang: you've banned members from this forum before for messy threads merely discussing apartheid. Surely advocating for war crimes — that the US itself considers crimes — is beyond the pale?
[dead]
israel will never be reasonable. we can bully them or appease them but they cannot be reasoned with.
appeasement is seemingly having the same effect it did when chamberlain did it.
Why should Israel have a right to exist? And under what parameters? Within which borders? Who gets citizenship?
Surely there's no moral case to be made for Israel having a right to exist in its current religious ethnostate form? People who presumably should have citizenship due to their ties to the land area are excluded because they believe in the wrong ancient delusions.
> Why should Israel have a right to exist?
1. Israel does not have a right to exist, in fact no state has "a right to exist".
2. Israel has a right to exist because her citizens successfully defend her from those who wish her not to exist.
3. Israel has a right to exist because the UN declared it.
>>> random.randrange(1,4)
3
> And under what parameters?
From the Israeli declaration of independence:
"WE EXTEND our hand to all neighbouring states and their peoples in an offer of peace and good neighbourliness"
- https://en.wikisource.org/wiki/Declaration_of_Independence_(Israel)
"No peace with Israel, no recognition of Israel, no negotiations with Israel."
- https://en.wikipedia.org/wiki/1967_Arab_League_summit
> Within which borders?
> Who gets citizenship?
In other words, there is no reason to treat Israel any differently than any other state on the planet.
> We attacked them
During negotiations, for the second time.
Negotiations during which the Iranians continuously stalled and continued their nuclear work. The threat of attack was part of the US negotiation strategy, and the Iranians thought they would call the "bluff". They were wrong.
Gentiles have a right to defend themselves.
Which is exactly why I expect the Americans to destroy the nuclear capability of the state that regularly chants "Death to America".
Did they chant that before or after hamas beheaded 40 babies? I lose track of all the lies.
[flagged]
They have been chanting "down with America" - that does not mean "murder every single person in America with their missiles (which can't reach America)"
"Death to" is a mistranslation of "marg bar", a phrase that is also applied to traffic, and inflation.
Do the Iranians want to kill all traffic and all... inflation?
We no longer believe your lies.
I'm grant you that I do not speak Persian, but I do speak Arabic and Hebrew. In Arabic the phrase موت لامريكا is common enough. And this Hebrew sign in Tehran says "prepare your coffins":
https://www.alamy.com/an-anti-israeli-mural-showing-a-launch...
So I do appreciate you educating me on the literal meaning of the Persian phrase, yet I dispute your interpretation that they state no intention of murdering us. Quite the opposite, the more I research it the more Hebrew banners in Tehran I see and I can conclude not only are they capable of murdering myself and my children, they have intent as well.
The primary condition is giving up nuclear ambitions.
Quid pro quo? Whoever requests that of others must do the same.
It is too late, Israel already has nuclear weapons.
[flagged]
Hasbara.
Over 20 years ago the president of Iran talked about "the regime occupying jerusalem must dissappear from the pages of time".
IE they want regime change in Israel - which is exactly what the Israelis want in Iran.
The difference is Iran is a more responsible international actor and has not started two wars based on this pretext.
So yes, I trust a nuclear armed Iran much, much more than I trust a nuclear armed Israel.
You'd trust a country that funds terrorist organizations with nukes? Let's not put you in charge...
Israel funded terrorist organisations in Syria, and in Palestine itself - most famously the group Hamas.
Many of the terrorist groups Iran funds operates in areas illegally occupied by the Israeli military, making them legitimate resistance fighters.
And Israel itself is a terrorist state - they achieved independence via the actions of Jewish terrorist groups in Palestine like Igrun, Lehi - which included several future Israeli Prime Ministers as members.
So no, I do not trust Israel with nukes - they should be disarmed immediately.
> Israel funded terrorist organisations in Syria, and in Palestine itself - most famously the group Hamas.
Last I checked Iran funded Hamas, and Hamas attacked Israel and Israel is trying to wipe out Hamas.
Also, Israel hasn't nuked anyone, so IMHO they can keep the nukes.
They should give the nukes to Hamas then. Oops, I dropped it.
Israel supports Hamas financially several times, a.) so they can justify a crackdown on Palestinians b.) to weaken the other political groups in Palestine that wanted to negotiate with Israel so only the most radical group is left to represent Palestinians, right wing Israeli assassinated the prime minister who negotiated a peace deal with PLO and the right wing is now in represented at top o government in Israel.
https://en.wikipedia.org/wiki/Israeli_support_for_Hamas
random sampling of newspaper articles about Israel's support of Hamas
https://www.jpost.com/israel-news/benjamin-netanyahu/article...
https://www.nytimes.com/2023/12/10/world/middleeast/israel-q...
> All these attempts to justify Iranian terror
At the end of the day you have to understand the reality that Iran is a sovereign nation that is going to defend itself. And yes they are hitting residential buildings and hotels with US military personnel present. None of this is terrorism, this is a nation state retaliating after an attack on their nation, you have to understand this basic concept, actions have consequences.
This is not propaganda, you are just willfully ignorant. If you want to destroy Iran you have to take retaliation into account, everything else is just propaganda, what do you expect them to do instead? Just lie down and take it?
You can't use retaliation of the nation you attacked as justification of why the attack was justified, its circular logic, this is textbook propaganda you are repeating.
I genuinetely do not think Hasbara like this works anymore. The overton window on this has irrevocably shifted since 2023 and it would be a better strategy for you to live within this new reality, rather than making ludicrous claims that the middle eastern country most vehemently trying to shape western views on the region is... Qatar. It just comes across as an obvious projection, and only encourages sentiment that has a real potential to become harmful to you personally.
That is, unless posts like thos are designed to encourage that sentiment, which I sometimes suspect.
I think this shifted overtone window has partially to do with why they started this war to begin with, they see the writing on the wall and their window of opportunity is closing. Trump is at historic lows in polling [1]; 65% of democrats now sympathize more with Palestinians over Israelis (17%) [2]. HN is just a generally reactionary place, I wouldn't read to much into that.
[1] https://www.natesilver.net/p/trump-approval-ratings-nate-sil...
[2] https://news.gallup.com/poll/702440/israelis-no-longer-ahead...
[flagged]
[flagged]
[flagged]
Instead of claiming "whatever bullshit you come up with" at me, go search for Qatari influence in English too. I prefer Qatari sources in Arabic because then one cannot claim a biased source, but for those who can not read Arabic there is ample English language discussion.
Here's just the first two Google results, enjoy, there's quite a few more from both sides of the US political divide if you would like to start nitpicking sources.
https://www.washingtonexaminer.com/news/3334519/elite-us-uni...
https://thehill.com/opinion/international/5334791-qatars-us-...
Everybody got flagged in this thread lol What is your argument again? Is it that Qatar used propaganda to make americans anti-war in the middle east or something? I don't even know at this point.
[flagged]
I believe they have warned that any country offering support will be targeted, even before the attacks began.
So they are cowards if they do what they say, and they are cowards if they don't do anything.
What should they do? Evacuate the country and offer the land for free?
> They assume that because their proxies (Hamas, Hezbollah) hide in civilian structures, so does the US army?
> Two US Defense Department employees were wounded when an Iranian drone struck a hotel in Bahrain's capital Manama, The Washington Post reported Monday.
Are you suggesting that two people sleeping in a hotel makes the hotel a valid military target? Because people have been telling Israelis for years that hospitals, mosques, residential apartment buildings, and schools from which rockets are stored and launched are not valid military targets, even when these activities are supported by the building administration and the rocket handlers are clearly visible.
I mean if you want to put your geopolitical blinkers on, sure...but how to beat America is old news at this point: cause casualties however you can, and wait for the US to give up.
Complain about it all you want but what are you going to do? The US is already bombing them.
Perhaps all of this goes into the big bucket labelled "war is expensive and unpredictable, maybe try diplomacy?"
Which the current administration has made a note of promptly tearing up prior agreements with everyone anyway so...whoops I guess.
[dead]
Comment was deleted :(
they have been engaging in hybrid warfare for decade+ they don't get to play victim this is the result of their continued proxy attacks
[flagged]
They can make it very expensive though. And they can't negotiate, given it's the second time they are attacked during negotiations, so really, what can they do? Cause the most chaos possible around them, strain the relationships between the US and other ME countries, force the US to make a choice about which ally to protect (it will be Saudi Arabia), make the oil price go up and deplete US weapon stocks. If the force the US and Israel to put boots on the ground, they will have won.
In WW-II, the US bombed the hell out of German forces for months on end. That is what people do not understand. The US have the capability to generate bombs indefinitely. There will be no boots on the ground for soldiers (they wish). They will just get pulverised as time goes by. If the US and Israel will think they cannot get to their thick skull they'll simply bomb the oil refineries and let the Iranian regime deal with paychecks from their street goons and fanatics who will eat them alive.
Its not bombs that are running out, its interceptors. The 12 day war exhausted 25% of american stocks, we're on day 3 of this round, do the math.
What happens to Israel when the interceptors run out and they're on equal standing with Iranian/Palestinian/Lebanese civilians?
> The US have the capability to generate bombs indefinitely.
The same US which had to re-build and re-open factories to be able to support Ukraine, and had an important shortage of shells for some time?
The same US talking with their allies to build ships for them?
US generals said that their defensive munition is not infinite. Middle Eastern countries said that they have Patriot stockpiles for 4 days.
We're past WWII. Nobody has that capacity anymore. Some of the tech and factories built these gigantic battle cruisers are not present anymore even.
US may, and can pulverize Iran if they want, but it'll be much more expensive than WWII era, because of how interconnected the world is now, and this is how post-WWII world has been designated. Make everyone depend on everyone, and make war very expensive as a result.
> I don't think it'll work, but they might as well try I guess.
Consider this from the eyes of the people living there. Your world is peaceful one day and burning tomorrow. It doesn't have to be "burning like hell", but something came from the sky, entered your building, exploded and damaged some stuff to the extent that fire-supression triggered and damaged more things.
Even if it's not a trauma, it's a shock. Something you'll be remembering for a long time. We live in fragile bubbles, but don't know it until we experience it pop. While this might not make them "win" the war, it'll leave a mark and make the affected persons' ears perch up to understand what's happening better.
Please note, I'm not from either side. I'm a close observer because of where I live, and still believe that this should have not happened.
Especially when 90% of the population are immigrants having no emotional ties to the ground
Yeah, it'll definitely trigger "why am I here and putting up with this" response in some people, and that's a breaking point for many of them.
Not just Amazon - I guess the Oil and gas industry is now run on the cloud. They used to have big SGI machines 30 years ago... but I bet everything is on the cloud now using GPUs.
I tend to believe that they still have their own clusters. For speed and privacy reasons. You don't want to give away the location of the oil you have found.
i have helped several major oil and companies migrate core infrastructure to the cloud.
plus, by the time you steal data relating to seismic surveys and reservoir analysis, you're years to late to exploit it as a competitor.
Thanks for the info. I stand corrected.
us govt and big business have always worked hand in hand, they compliment each other.
They complement each other too, which is more impactul than any compliments they might send.
Some may even claim it's impactful.
you never know, people love to be loved.
In the eighties my friends and I used to think we would be the first to go in a nuclear strike because we were close to an American air base. Now I have to worry about living close to an Amazon data centre.
Comment was deleted :(
Lesson learned: If your recovery plan requires calling any API in the dead region — to detach an IP, describe a route table, launch an instance, read an S3 object, or decrypt a volume — it will fail when you need it most.
Every dependency on the primary region is a dependency on the thing that just broke.
Hmmm. We’ve seen the fun that comes from cutting data cables and pipelines. Think that’s been factored in with the asymmetric warfare coming from the Middle East? Perhaps some network assaults as well?
Krugman has pointed out that modern war is bloody expensive. Perhaps resistance will just be helping burn money? Lots of motivated people on one side. And I hope countries are being careful, as a Thirty Years War in the Middle East would suck.
the DR test isn't 'can we run in region B.' it's 'can we cut over to region B when every API call to region A returns a timeout.' most recovery plans assume they can still reach the thing that just broke
Related discussion: https://news.ycombinator.com/item?id=47209781
Any chance this is causing the claude issues directly/indirectly?
Most people assume its because of the influx in chatgpt users switching to claude. 295% daily uninstall hike of ChatGPT: https://techcrunch.com/2026/03/02/chatgpt-uninstalls-surged-...
Why would they funnel all their traffic to a middle eastern AZ?
Capacity is tight, you serve from where you can.
Probably also because most token use cases are not latency sensitive. A 200ms extra delay isn't going to change much for most use cases.
Right, so if they were able to get a discount in UAE…
> U.S. Military Using Anthropic for Middle East Airstrikes[1]
> Strikes hit AWS datacenter
> Antropic aided strikes causes Anthropic outage lol
[1] https://www.wsj.com/livecoverage/iran-strikes-2026/card/u-s-...
This comment makes it seem as if it's the US who striked the Amazon data center.
The US lost 3 of its own fighter jets yesterday due to friendly fire[0], is it out of the realm of possibility?
[0] https://newrepublic.com/post/207197/us-shoots-three-fighter-...
US tax dollars blowing up US tax dollars. Lockheed Martin and Boeing stonks go up.
Generally speaking defense contractors do not make more money during wars, because we start asking them to make low-margin realistic stuff, whereas in peace time we get bored and start paying them to design imaginary superweapons.
But we also usually don't blow up our own planes I suppose.
>Generally speaking defense contractors do not make more money during wars
US economy is not in war mode right now in order to ask its contractors to make cheap stuff.
It wasn't the US attacking the UAE, just FYI.
I find it more likely that Anthropic has rented space in other data centers in the region that were also impacted by Iranian retaliation
Typical BBC reporting: Amazon's cloud computing business says drones have hit three of its facilities in the United Arab Emirates (UAE) and Bahrain following US and Israeli strikes against Iran at the weekend. The incidents occurred on Sunday morning, with Amazon Web Services (AWS) saying at the time that ''objects'' had hit a data centre in the UAE, creating ''sparks and fire''. Also on Sunday, AWS said it was investigating power and connectivity issues at a facility in Bahrain. On Monday, the company confirmed that drone strikes had caused the outages.
Notice how they do not mention that the facilities were damaged by Iranian attacks on the UAE and Bahrain but following US and Israeli strikes against Iran at the weekend.
Somewhat OT but it remains remarkable how the knee-jerk down-vote-button brigade feels the need to vote down a totally unrelated post on getting a refund for a Microsoft Home Server [1] and one on the relation between hardware + low-level systems software capabilities versus applications software just because I happened to voice an opinion outside of their desired narrative. Grow up, people. If your opinions are so weakly founded that you feel the need to 'punish' those who dare to voice dissent you should get some more soundly founded ones.
First outage postmortem with actual mortem.
Crafted by Rajat
Source Code