The two man rule

Let’s say we have a big company with lots of servers, providing services to lots and lots of customers. Where should they store their owns passwords? A centralized system like LDAP and enforcement of 2-factor authentication is appropriate but what about the encryption keys for financial data? What about administrative overrides for network equipment? If the network is down, how can you let people log in to a router to fix it if the router can’t check the user logging in against the LDAP server? We need break glass accounts and the like.

So there will need to be some repository of sensitive data that is rarely needed. I think we can print it on paper! It has very predictable deterioration properties, unlike electronic devices that can go from 100% functional to 100% completely fried in a nanosecond due to an unfortunate cosmic ray hitting just the wrong atom*. It’s also highly resistant to hackers. The Israeli intelligence service is said to keep their most sensitive data on paper only.

But do we really want to keep these super-admin overrides on a bunch of paper where someone can just flip through them and take pictures of them with their smart phone? No, that would be bad. I have a proposal:

I’d recommend generating these kinds of passwords for sets of devices. You buy ten new switches? Generate admin passwords for those ten and print them out on paper. Two papers. The first half of each password goes on one page. The other half on another. The people doing the work can check that the two pages contain the complete passwords so we know they will work when needed. It’s kind of hard for people to remember ten 20-character random passwords that they’ve seen once, so it’s not a big security issue that the passwords are checked. We want to balance security with reliable access in an emergency.

Now things start to get a bit administrative. I suggest each paper is placed in a sealed envelope with a tamper evident label. I bought some from Amazon just to try it out:

Before I “opened” it the blue color was uninterrupted. Afterwards the white text appeared(I think it’s supposed to say “Void, opened”) and I’d struggle to get the things back to it’s original condition. Note the serial number. I wouldn’t trust the manufacturer to never reuse serial numbers. But it’s not in their interest to offer these labels to people with whatever serial numbers the customer asks for, so it’s not trivial to get your hands on duplicates. You’d probably have to buy an excessive number of labels to have a chance of getting duplicates.

Illustrations with a single password for simplicity:

Now we have our password-halves in sealed envelopes and we can lock them away. Preferably in two locations where no one person has access to both. And the password could only be used with more than one participant involved. The use case would be something like this:


Bob: Uhm, boss?
Boss: Yeah, what's up?
Bob: Uhm, we need to access the snapshots and binlogs for the database server that handles financial transactions.
Boss: What happened?
Bob: A bad patch was rolled out that deleted a bunch of stuff. We need to restore the data from snapshots and a binlog replay.
Boss: Okey, do it!
Bob: We need your help.
Boss: How? I don't even know what a binlog is. I've heard you talking about it but that's it.
Bob: Well, it turns out the normal decoding stuff isn't working. You know, with the hardware tokens?
Boss: Sounds vaguely familiar. Oh, now I get it! We need the override password from the safe?
Bob: Well, two safes. One half is in one safe, the other half is in another safe. Here's a write-up of what we want and why.
Boss(reads paper): Okey, but we need one more person here.
Bob: Oh, right! Jim, get in here!
Jim: What?
Bob: We need you to sign the paper. It has to be two people asking for the override before we can go ahead.
Jim: All right, what do I need to do?
Bob: Do we need the override password for the snapshots and binlogs of the financial transaction server?
Jim(looking confused why the question is being asked): Yes...
Bob: Then sign on the dotted line here.

Jim signs.

Boss: Good, then I'll sign it and we can get cracking.

Department A:
Bob: Hi! We need the ledger over secure envelopes in department B.
Boss(whispering to Bob): Why doesn't the ledger at department A cover their own envelopes?
Bob: Because then they could fiddle with both the ledger and the envelopes and we would none the wiser. If department B wants to open an envelope and edit the ledger to hide that transgression, they need help from department A.
Boss(thinking): Okey, makes sense. Also, you could just call it "rule breaking" rather than "transgression". It would be easier.

Bob shrugs.

Bob(grabbing ledger): Thanks!

Department B:
Bob: Hi! We need a sealed envelope... (looking in ledger). Number 1203.
Harry: Okey. Hi boss!
Boss: Hi.
Harry: You have the paperwork?
Bob: Yeah, here it is.
Harry: Good. I'll just get and fetch the envelope.
Bob: Oh, we need the ledger for envelopes for department A.
Harry(looking at the paperwork again): Sure, I'll get that too.

Harry comes back with the two items.

Harry: May I have the ledger?
Harry(talking to himself, comparing records): 488901 from 2023-04-22. Yeah, that's good. I'll just cross it out in the ledger. There!
Bob: Can I see it now? I'll need to write down password 7.
Harry: Here you go.
Bob: Thanks, got it. Will you seal it back up?
Harry: Yes, I'll get a new label and envelope.

Harry seals the envelope with a new label.

Harry: Label number 489102 and today is 2023-07-03. I'll sign the label and add a new ledger entry.

Bob and his boss look on as things are prepared.

Harry: Done.

Bob: Good, we'll be back shortly with the ledger.

Department A is the same thing. Talk to someone who can fetch the paper, check that no one has accessed the envelope since the last authorized access, etc. The team then returns the ledgers they got from the two departments.

Bob, Jim and their boss all go to a workstation and log in using the secret password. Bob fixes the problem and logs back out again. They all go to a paper-shredder and destroy the copy of the password. The signed document authorising access to the envelopes is marked as "VOID" using a stamp. Job's done!

To be really secure the system that the password grants access to should have a new password set. Several people have seen the current password after all. But this would require printing out a new pair of papers, getting rid of the old ones and so on. For a bank this might be worth the effort and then there should probably be just one password(half) per envelope. For a tech company the simplification shown above is perhaps more appropriate.

I originally conceived of this system when I was thinking about how it could come to pass that a single systems administrator at the NSA could leak thousands of documents. No one person should have that kind of access to sensitive data.

If circumventing access control at the NSA had required a password split up between two safes, using tamper-evident serialized labels recorded in a ledger held by someone else than the custodian of the envelope, I think the leak would have been much less likely. Now, this system is slow and cumbersome. It would never fly for anything approaching standard password management. But for “the keys to the kingdom” at a service provider or a government agency? It might be worth the hassle.

The inspiration is the American two man-rule for handling nuclear weapons which states that such a weapon can’t be left with any one person. Locking it away in a vault? That’s okey. Having two people wheel one out to a waiting bomber? That’s, okey. Letting Bob watch the nuke while Jim goes in to fetch his mittens? No, it’s the two man-rule. I would be remiss for not recommending the Always/Never series about the development of various technologies developed to make handling nuclear weapons more secure.

The tamper-evident label-thing is inspired by how codes on nuclear submarines are stored and handled. Or how they are allegedly stored and handled, it’s not like the US navy is doing show-and-tell of exactly how a nuclear weapons launch would work. One thing that I’m thinking about is if envelopes can be resealed if you cut them open at the sides. I watched a video about research done by a US lab working with nuclear materials on tamper evident packaging and I don’t think they’d be impressed with “this envelope doesn’t look like it was cut open and then resealed using wet paper, heat and some pressure. See for instance this document published by the IAEA about testing various storage devices tested by Sandia: https://inis.iaea.org/collection/NCLCollectionStore/_Public/27/020/27020417.pdf

Sandia was also heavily involved in nuclear weapons security improvements as shown in the documentary series mentioned earlier. It’s a subject of some importance as can be seen in this short video which also covers civilian use of nuclear fuel and how to avoid diversion of nuclear materials.

Anyway, metal foil seems like a reasonable alternative to paper envelopes. It appears quite difficult to reseal without it being evident. By the way, tamper evident is preferred over tamper resistant because the latter isn’t feasible. It’s better to say “yes, with enough time any barrier can be overcome but we’ve made sure that it’s obvious that someone has opened the item”.

Games I like and some that I don’t

Infinitely replayable

  • Fallout 3
  • Mass Effect
  • Mass Effect 2
  • Final Fantasy VII (PSX)
  • Civilization IV
  • XCOM: Enemy Unknown/Enemy Within
  • Frozen Synapse
  • BookWorm Adventures Volume 2
  • Minecraft Tekkit
  • Super Mario 64 (Nintendo 64) – I don’t think people without nostalgia for the game will like it.

Good

  • Assetto Corsa
  • Braid
  • Fez
  • Far Cry 3
  • Far Cry Blood Dragon
  • SpaceChem
  • Prince of Persia: The Sands of Time (PS2)
  • Chained Echoes
  • Factorio
  • Subnautica
  • The Talos Principle
  • Tropico 4
  • Viscera Cleanup Detail
  • Creeper World III: Arc Eternal
  • Bioshock
  • Bioshock Infinite
  • Shadowrun Returns
  • Kerbal Space Program
  • Fallout: New Vegas
  • Hotline Miami
  • Batman: Arkham City
  • The Swapper
  • Metroid Prime 3: Corruption (Wii)
  • Advance Wars (Gameboy Advance, series)
  • Super Mario Galaxy (Wii, series)
  • Door Kickers
  • Half-life
  • Half-life 2
  • Portal
  • Portal 2
  • Red Faction: Guerilla
  • World of Goo
  • Trials 2: Second Edition
  • Trials Evolution Gold Edition
  • Rush
  • Hexcells (series)
  • The Legend of Zelda: Ocarina of Time (N64)
  • The Legend of Zelda: A Link to the Past (SNES)
  • The Legend of Zelda: Twilight Princess (Wii) – The weakest Zelda game I’ve played that wasn’t Skyward Sword bad.
  • Mark of the Ninja
  • Nuclear Throne
  • Huntdown
  • Trine (series)
  • Operation Flashpoint: Dragon Rising
  • Simutrans/OpenTTD
  • Super Mario Sunshine (Gamecube)
  • God of War (PS2)
  • Deus Ex: Human Revolution
  • Ratchet & Clank (PS2)
  • Resident Evil 4 (PS2)
  • Grand Theft Auto III (PS2)
  • Grand Theft Auto: Vice City (PS2)
  • Metal Gear Solid (PSX)
  • GoldenEye 007 (N64)
  • Super Metroid (SNES)
  • Infested Planet
  • Mafia (2002)
  • Metro 2033
  • Gunpoint
  • Prison Architect
  • The Legend of Heroes – Trails in the Sky First Chapter

Candidates

  • Stardew Valley
  • FTL
  • Psychonauts
  • Darksiders Warmastered Edition
  • Cities: Skyline
  • Battletech
  • Children of a Dead Earth
  • Kingdoms of Amalur: Reckoning
  • Astlibra Revision
  • Satellite Reign
  • TIS-100
  • Ys-series
  • Tales of-series
  • Octopath Traveller
  • Dyson Sphere Program
  • CrossCode
  • Homeworld Remastered Collection
  • Shadowrun: Dragonfall
  • Ys Origin
  • The Legend of Heroes: Trails in the Sky
  • Torchlight II

Not right now

  • Bravely Default II – Visually overwhelming and not keen on the music. Otherwise it seems good.
  • Atelier Sophie – JRPG with 3D graphics might not be for me. I like the world of this game so I’ll try it again later.

No

  • Fallout 4
  • Disgaea
  • Primordia
  • The Bureau: XCOM Declassified
  • The Elder Scrolls IV: Oblivion

Things that we should all pretend never existed

  • Mass Effect 3
  • Legend of Zelda: Skyward Sword (Wii)
  • Metal Gear Solid 2 (PS2)

BookWorm Adventures Volume 2

I play this game from time to time but it needs to be played in moderation lest my brain gets locked into “letter permutation mode” which isn’t great. Big ups to Extra History on YouTube for bringing me the word debentures, a type of IOU issued by the British armed forces back in the day.

Sokrates ’63

ASAC: Come in!
Agent: Sir! Have you heard?
ASAC: About the president? Yes, I heard he died after being shot. It's been all over the news.
Agent: I'd like to volunteer to go down to Dallas to start investigating the assassination!

The agent stands at attention looking straight ahead, over the head of his boss.

ASAC: Could you close the door?
Agent: Uhm... Yes, of course!
ASAC: Thank you! Now, I have a few questions...

The agent in charge frowns and looks around the office.

ASAC: Could you tell me what leads you will investigate in Dallas and what you will uncover?
Agent: ... No, sir. I mean, I can tell you that we would start checking the physical evidence and work our way out from this suspect named Oswald, but where the investigation would go from there is something that we will have to figure out along the way. That's how things always work.
ASAC: Mmm... Let me put it slightly differently, can you tell me right here and now that your investigation won't under any circumstances investigate employees of the US federal government?
Agent: No, that would be precisely the sort of thing that we can only answer after the fact.

The agent in charge smiles.

ASAC: Ah, but that's just the thing! You would have to give that guarantee to be part of this investigation.

The young agent looks confused.

ASAC: Look, this just happened a few hours ago and I haven't heard word one from the top brass about this, but I can tell you what you should expect. No one in a position of authority is going to let an investigation of the murder of the president of the United States so much as imply that the military or the CIA was involved.
Agent: Do you really think they are?!
ASAC: That's not the point. The questions will be asked - whether it be born of a genuine suspicion or a wish to embarrass the government - and it will be the responsibility of investigators to categorically rule out any such connection. The Cuban missile crisis last year almost set the world on fire! Do you think the Soviet Union would allow there to be so much as a sliver of indication that the military is trying to take control?
Agent: ...
ASAC: The US can't afford to spend six months with half the government eyeing the other half under suspicion of arranging a coup d'état. I hope someone will investigate this assassination properly and... deal with whoever was behind it. But the FBI won't be doing that work. Our job will at most be to rubber-stamp a finished product that someone higher up the food-chain thinks is geo-politically acceptable. Do you want that job? Did you come in here to volunteer signing whatever whitewash the "powers that be" decides on?
Agent: No.
ASAC: I didn't think so. You want to do what I want to do, you want to investigate properly no matter where the investigation goes. But that's not on the cards. So please... Go back to work. Keep doing what your're doing and keep your nose out of this.
Agent: I see. I appreciate that you cleared this up.
ASAC: Politics is always a part of what we do, but it doesn't usually override the mandate congress has given us or the laws of this land. Here? It will.

The agent nods at his boss and turns to leave. He stops before he reaches the door.

Agent: Boss? Can I ask you one more thing?
ASAC: Sure.
Agent: What if they catch Oswald? Alive, I mean. He might point fingers every which way. Won't someone have to investigate those leads at that point?
ASAC: Uhm... I hadn't thought of that. Let's put it like this: if Oswald is arrested and he makes accusations against the military or some big-shot politicians... We're never going to hear about it. If there are ever transcripts of his testimony made public they will be very innocuous and Oswald will not challenge them. Perhaps because he won't be able to challenge those records. I'll bet you a steak dinner!
Agent: I understand.

The point of this scene comes from a thought-experiment: why did the FBI, the Dallas Police and the Warren commission investigate the assassination of John F. Kennedy so poorly? The quality ranges from “Not super-interested in hearing new things” in the case of the Warren commission to “Oliver and Hardy” in the case of the Dallas Police. It’s not entirely unreasonable to draw the conclusion that it was a massive cover-up to hide some deeper conspiracy. But my argument here is that no one doing any kind of official investigation was willing to ask certain questions simply because those questions were too dangerous, independently of the answer.

Simiarly you are unlikely to ever hear the head of the SEC or FDIC call for an investigation into whether the largest bank in America has falsified their records to hide insolvancy. That would crash the economy in about thirty minutes. So it’s just not done. It’s the sort of thing that is handled behind the curtain and if things are really bad the government announces the problem at the same time as their bail-out to keep the country from falling apart.

That the cover-up wasn’t done because of some specific knowledge of who was involved doesn’t make the cover-up any less of a cover-up. It’s just that we can’t necessarily draw the conclusion that the FBI knew that the CIA was involved in the assassination. It all had to point squarely at one guy and maybe some punks that guy knew. No links to his time in the military. No questions how he defected to the Soviet union and got help coming back to the US after two years instead of being investigated for treason. It was just one guy. The FBI probably started typing up that stuff the same day as the assassination.

Personally I think the ham-fisted investigations of the JFK assassination actually serves as something of an argument against it being a properly organized hit involving the CIA and the DoD. “The only suspect in the killing of the US President was just shot dead while in police custody, having been questioned without a lawyer present for two days leaving all statements inadmissable in a future court proceeding against him.”

In the movie JFK(which I recommend watching but which should also be taken with a grain of salt) the protagonist asks something along the lines of:

If you heard Premier Chrustchev was shot dead while being driven through Moscow, and that the only suspect was a capitalist who spent two years living in the US, would you take that a face value? When the only suspect is shot and killed while in police custody, do you not sense that something might be off? When law enforcement concludes almost immediately that there was one shooter and absolutely no one else was involved, do you consider that to be reasonable?

As competently as the assassination was performed, so the cover-up was incompetently performed. I assume Oswald was meant to be killed in a shoot-out with the police but that didn’t quite work. Seems like it would have been worth the effort to do that job properly. It was a mess when Oswald was captured alive. Now he had to meet an untimely end in police custody and in the meantime his statements created a big headache. His words “I am a patsy!” were caught on a TV news camera and is to JFK assassination conspiracy theories what the phrase “Let there be light!” is to the bible. Whoever was involved knew how to shoot really well and not get caught but to line up all the puzzle-pieces to make a nice believable story about a jackass named Oswald – who couldn’t do anything right – killing the president all by himself? Not so much.

Twitter in flames

As someone who considers Twitter to be equivalent to a disease I’m entertained by Elon Musk burning the company to the ground. He’s like Tony the security guard in this RiffTrax short about Hormel.

Tony has the Guinness world record for Most wrong decisions made in a lifetime.

Firing staff only to realize that some of them were necessary to maintain operations wasn’t great.

Blocking links to rival social media platforms is quite astounding. Something tells me that agencies like the FTC or the Swedish Competition Authority will take exception to that kind of move. It also goes down very poorly with netizens, which seem like an important part of the demographic for Twitter.

Now Twitter is issuing new shares to stay afloat.

In comes the EU to make the situation even worse. In writing this I’m reminded of JonTron’s attempt at being politically correct in his criticism of a bad game. The EU arguing that reporters’ ability to use a privately owned and operated service is an important part of media freedom is quite embarrassing. The remarks are not made more reasonable by the implication that a reporter has greater rights than non-reporters. The EU don’t condemn arbitrary suspensions of ordinary Twitter accounts, just those of reporters or politicians. It would have been nice if they had at least pretended to uphold egalitarian principles.

Not that it was a clever move for Twitter to ban the reporters in the first place. On account of issues faced by Elon Musk personally Twitter now takes the view that providing information about people’s real-time location is “doxxing”:

Any account doxxing real-time location info of anyone will be suspended, as it is a physical safety violation. This includes posting links to sites with real-time location info.

Elon Musk quoted by the BBC | https://www.bbc.com/news/world-us-canada-63978323

While restrictions of publishing people’s real-time location isn’t entirely unreasonable the ElonJet-account simply tweeted publicly available information about the movement of airplanes. To then go on a banning-spree of reporters who write about the suspension of the ElonJet-account and blocking links to Mastodon…

Can we now start talking about Hyperloop being a boondoggle from the very start and that Tesla has been promising more than it can deliver? Paypal and SpaceX are great accomplishments and Tesla has had a net-positive impact on the world(even if it makes promises it can’t keep) which Musk should be given credit for. But people shouldn’t trust his judgement in all matters more than they trust their own common sense.

2022-12-19

Well that escalated quickly. First banning people for referencing their accounts on other social media platforms: https://www.svt.se/nyheter/utrikes/twitter-forbjuder-lankar-till-konkurrenter [ In Swedish ]

And then Elon holding a public vote on whether he should step down as head of Twitter: https://www.bbc.com/news/business-your-money-64021412

Credit to him for taking complaints seriously, but I think it would be sufficient for him to run major changes by some people before rolling them out. You’ve got employees, most of them are probably at least of average intelligence. Why not avail yourself of their recommendations? You don’t have to abide by them. But listening to these questions carefully would have served Elon well these past few weeks:

  • Are we sure all these employees are redundant? Maybe we should understand the company a bit more before we fire half the workforce?
  • Banning people from linking to competitors might not be great. Don’t we risk picking a fight with the EU at the very least? The US DoJ might not be super-happy with it either…
  • Should we really be talking about freedom of speech in such black-and-white terms? It’s kind of a tricky issue – it has been for centuries – so saying that we’re for free speech is going to lead to lots of misunderstandings and accusations of not living up to our commitments. Because everyone has their own idea of what “free speech” is exactly.

Helpful Pointers for the British Conservative Party

In the interest of fairness I can’t tear Russia’s idiotic decisions to shreds and not call out the British Conservative Party for using the United Kingdom as a stage on which to play out their reality-TV “drama”. They spent a year bickering about kicking Boris Johnson out, then they finally did, then they spent three months choosing a new one who spent a month and a half in office before being forced to resign by her own party. If they don’t announce that a new general election will be held they might as well change the British national anthem to Einzug der Gladiatoren.

You can’t drag the whole country down with you as you play Survivor. Well, I mean you obviously can do that but I mean that you at least have to give the electorate an opportunity to vote on whether they accept your attempt to change the form of government from constitutional monarchy to Benny Hill slapstick. Because you didn’t run on a platform of Benny Hill slapstick in the previous election! You gave no indication whatsoever that you were intending to treat the positions of prime minister or chancellor of the Exchequer as a game of three card Monte.

The thing is, even if a very competent government with ample support from it’s own party ran the country they’d still be hard pressed to improve matters. Because the UK has been racking up massive debt for a long time and so it can’t borrow lots of money to introduce reforms that are necessary for the country to improve its ability to pay back the money it owes. That’s why you don’t rack up quite so much debt! It leaves you with a lot fewer choices.

If they cut public spending on things that don’t generate income, focus heavily on education and expand both IT and traditional industry domestically they might be able to crawl up the hill they are currently sliding down. The services they cut may well be things that improve public health, standard of living and international prestige but right now the UK is fighting to stave off bankruptcy. When you reach that point you need to play hard ball. Again, this is why you don’t decay for years and years and years and just say “This is fine.

2022-10-22

So far, the BBC estimates that Mr Sunak has the most MPs declaring their support, with 63, compared to 38 for Mr Johnson and 17 for Ms Mordaunt.

https://www.bbc.com/news/uk-politics-63338171

I often think about what Yahtzee of Zero Punctuation said about people launching a new MMO:

So the obvious question when you’re proposing a new MMORPG is “ARE YOU FUCKING INSANE?! World of Warcraft picks bigger rivals than you out of its teeth every morning!

https://youtu.be/FRXb1Lr4G3c?t=107

Someone needs to get the Tories together and run all their ideas through Yahtzee or some equivalent before saying anything publicly. I thought the worst case scenario here was that they elected someone new and then fired them after six hours. It never occurred to me that bringing back Boris Johnson would even be part of someone’s fever-dreams. You spent a year publicly debating his position as party leader before finally booting him out! That was fifteen weeks ago! I’m not surprised that he has his supporters within the party but I had thought some modicum of common sense would keep people from discussing this where other people could hear them…

2022-10-25

So Boris Johnson bowed out and Rishi Sunak won. Credit to Boris Johnson for seeing the absurdity in him vying for the leadership a few months after being kicked out. Now begins the real work, as opposed to Tory drama and picking fights with Britain’s creditors! I suspect Mr Sunak will go down in infamy. He will have to bring reality to the British people and I don’t think they will like that. He will fix the country and will be blamed for all the hardship that his predecessors made an inevitability. A sin-eater.

Favorite MST3K episodes

  • K09 – Phase IV
  • K19 – Hangar Eighteen
    It has a great scene with Robert Vaughn around 00h:15m-00h:20m where they arrange a cover-up about a UFO and reason back and forth about the hows and the whys. Actually a movie worth watching in its own right. Basically Capricorn One with some X-Files thrown in.
  • S01E02 – The Robot vs The Aztec Mummy
  • S01E13 – The Black Scorpion
  • S02E12 – Gozilla vs Megalon
  • S03E06 – Time of the Apes
  • S03E10 – Fugitive Alien I
  • S03E14 – Mighty Jack
  • S03E18 – Star Force Fugitive Alien II
  • S03E21 – Santa Claus Conquers the Martians
  • S03E23 – The Castle of Fu-Manchu
  • S04E20 – The Human Duplicators
  • S04E22 – The Day the Earth Froze
  • S04E23 – Bride of the Monster
  • S04E24 – Manos the Hands of Fate
  • S05E01 – Warrior of the Lost World
  • S05E21 – Santa Claus
  • S06E02 – Invasion USA
  • S06E06 – The Creeping Terror
  • S06E08 – Codename Diamond Head
  • S06E17 – The Sword and the Dragon
  • S06E19 – Red Zone Cuba
  • S06E20 – Danger Death Ray
  • S07E01 – Night of the Blood Beast
  • S07E04 – The Incredible Melting Man
  • S07E05 – Escape 2000
  • S08E01 – Revenge of the Creature
  • S08E04 – The Deadly Mantis
  • S08E06 – The Undead
  • S08E08 – The She Creature
  • S08E10 – The Giant Spider Invasion
  • S08E13 – Jack Frost
    Finnish or Russian movie about some deity that creates winter.
  • S08E14 – Riding with Death
    Two episodes from an intended TV series glued together to make a TV-movie. Nicely 70’s flat and stupid.
  • S08E15 – Agent for H.A.R.M
  • S08E16 – Prince of Space
    Atrocious Japanese black-and-white sci-fi. Marvelous MST3K-episode.
  • S08E17 – Horror of Party Beach
  • S08E18 – Devil Doll
  • S08E19 – Invasion of the Neptune Men
    Another atrocious Japanese black-and-white sci-fi. Another marvelous MST3K-episode.
  • S08E22 – Overdrawn at the Memory Bank
  • S08E20 – Space Mutiny
  • S09E01 – The Projected Man
  • S09E03 – Puma Man
  • S09E05 – The Deadly Bees
  • S09E06 – The Space Children
  • S09E07 – Hobgoblins
    A movie so blisteringly awful it almost makes the inventor of the movie camera renounce his great work from the grave.
  • S09E09 – Gorgo
  • S09E11 – Devil Fish
  • S10E04 – Future War
  • S10E05 – Blood Waters of Dr Z
  • S10E07 – Track of the Moon Beast

Ceph-csi and Nomad

I have three manager-nodes in a proof-of-concept cluster: manage01, manage02 and manage03. There are also three worker nodes: worker01, worker02 and worker03. Managers are identical to each other, as are workers. Manage-nodes have Ceph mons. Worker nodes have Ceph OSDs. Each configured with the same ansible collection. There are of course some differences. Different IP addresses, different hostnames, they joined the cluster at different times but that shouldn’t matter and it turns out it doesn’t.

But what has happened is that only worker01 and worker02 have been able to mount Ceph RBD-volumes via ceph-csi. Isn’t that weird? Surely three identical(with the caveats above) servers should behave the same? I’ll skip to the answer to the riddle: ceph-csi in node-configuration running as a Nomad system-job can not connect to an OSD on the same host. worker01 always mounted an RBD volume by connecting to an OSD on worker02 or worker03. Worker02 always connected to an OSD on worker01 and worker03. But worker03 always tried to connect to its own OSD, which just fails by timeout. When I deleted the OSD on worker03 everything worked just with on worker03. It has no trouble mounting an RBD volume.

As we will soon see, this isn’t a Nomad or ceph-csi-specific issue. Yes, I said “ceph-csi in node-configuration running as a Nomad system-job can not connect to an OSD on the same host” but that’s still true. It’s just not the only situation in which the rbd command fails to mount an OSD on the same host. It’s probably a very unusual configuration to have ceph-csi node-instances running on the same host as OSDs. So I just co-located OSDs with mons on the manage-nodes and everything works fine.

Now to my point: why does almost everything aimed at running as/with containers have such bad debugging support? Let’s look at what happens when ceph-csi can’t mount an RBD volume because it tries to connect to the OSD located on the same node:

I1005 22:26:42.857254       1 utils.go:212] ID: 115 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":5}}}]}
I1005 22:26:45.963342       1 utils.go:195] ID: 116 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC call: /csi.v1.Node/NodeStageVolume
I1005 22:26:45.963873       1 utils.go:206] ID: 116 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC request: {"secrets":"***stripped***","staging_target_path":"/local/csi/staging/ceph-mariadb-test06/rw-file-system-single-node-writer","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"c7a90a82-4109-11ed-9a14-7b981f44cc3b","imageFeatures":"layering","imageName":"csi-vol-7403da80-fa1c-44e6-9794-2dca4ed3fbd7","journalPool":"nomadstore","pool":"nomadstore"},"volume_id":"0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7"}
I1005 22:26:45.964612       1 rbd_util.go:1279] ID: 116 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 setting disableInUseChecks: false image features: [layering] mounter: rbd
I1005 22:27:12.859386       1 utils.go:195] ID: 118 GRPC call: /csi.v1.Identity/Probe
I1005 22:27:12.859389       1 utils.go:195] ID: 118 GRPC call: /csi.v1.Identity/Probe
I1005 22:27:12.859995       1 utils.go:206] ID: 118 GRPC request: {}
I1005 22:27:12.859998       1 utils.go:206] ID: 118 GRPC request: {}
I1005 22:27:12.860186       1 utils.go:212] ID: 118 GRPC response: {}
I1005 22:27:12.860187       1 utils.go:212] ID: 118 GRPC response: {}
I1005 22:27:12.860962       1 utils.go:195] ID: 119 GRPC call: /csi.v1.Node/NodeGetCapabilities
I1005 22:27:12.861096       1 utils.go:206] ID: 119 GRPC request: {}
I1005 22:27:12.861634       1 utils.go:212] ID: 119 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":5}}}]}
I1005 22:27:42.862239       1 utils.go:195] ID: 120 GRPC call: /csi.v1.Identity/Probe
I1005 22:27:42.862341       1 utils.go:206] ID: 120 GRPC request: {}
I1005 22:27:42.862407       1 utils.go:212] ID: 120 GRPC response: {}
I1005 22:27:42.862550       1 utils.go:195] ID: 121 GRPC call: /csi.v1.Identity/Probe
I1005 22:27:42.862578       1 utils.go:206] ID: 121 GRPC request: {}
I1005 22:27:42.862598       1 utils.go:212] ID: 121 GRPC response: {}
I1005 22:27:42.863718       1 utils.go:195] ID: 122 GRPC call: /csi.v1.Node/NodeGetCapabilities
I1005 22:27:42.863834       1 utils.go:206] ID: 122 GRPC request: {}
I1005 22:27:42.864018       1 utils.go:212] ID: 122 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":5}}}]}

Mhm… So… It gets to setting disableInUseChecks: false image features: [layering] mounter: rbd and then just nothing? What happens when thing work?

I1006 13:43:56.363432       1 utils.go:212] ID: 46 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":5}}}]}
I1006 13:43:56.984542       1 utils.go:195] ID: 47 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1006 13:43:56.984866       1 utils.go:206] ID: 47 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC request: {"target_path":"/local/csi/per-alloc/08a19d7d-fb2c-8f36-6a79-84acb6324c72/ceph-mariadb-test06/rw-file-system-single-node-writer","volume_id":"0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7"}
I1006 13:43:56.986804       1 utils.go:195] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC call: /csi.v1.Node/NodeStageVolume
I1006 13:43:56.987341       1 utils.go:206] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC request: {"secrets":"***stripped***","staging_target_path":"/local/csi/staging/ceph-mariadb-test06/rw-file-system-single-node-writer","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"c7a90a82-4109-11ed-9a14-7b981f44cc3b","imageFeatures":"layering","imageName":"csi-vol-7403da80-fa1c-44e6-9794-2dca4ed3fbd7","journalPool":"nomadstore","pool":"nomadstore"},"volume_id":"0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7"}
I1006 13:43:56.989185       1 rbd_util.go:1279] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 setting disableInUseChecks: false image features: [layering] mounter: rbd
I1006 13:43:56.989833       1 utils.go:212] ID: 47 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC response: {}
I1006 13:43:56.993063       1 utils.go:195] ID: 49 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC call: /csi.v1.Node/NodeUnstageVolume
I1006 13:43:56.993150       1 utils.go:206] ID: 49 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC request: {"staging_target_path":"/local/csi/staging/ceph-mariadb-test06/rw-file-system-single-node-writer","volume_id":"0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7"}
E1006 13:43:56.993231       1 nodeserver.go:919] ID: 49 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 an operation with the given Volume ID 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 already exists
E1006 13:43:56.993268       1 utils.go:210] ID: 49 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 already exists
I1006 13:43:57.059403       1 omap.go:88] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 got omap values: (pool="nomadstore", namespace="", name="csi.volume.7403da80-fa1c-44e6-9794-2dca4ed3fbd7"): map[csi.imageid:d37e95b82f3a csi.imagename:csi-vol-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 csi.volname:ceph-mariadb-test06]
I1006 13:43:57.100591       1 rbd_util.go:346] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 checking for ImageFeatures: [layering]
I1006 13:43:57.183197       1 cephcmds.go:105] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 command succeeded: rbd [device list --format=json --device-type krbd]
I1006 13:43:57.208985       1 rbd_attach.go:420] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 rbd: map mon manage01,manage02,manage03
I1006 13:43:57.362451       1 cephcmds.go:105] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 command succeeded: rbd [--id admin -m manage01,manage02,manage03 --keyfile=***stripped*** map nomadstore/csi-vol-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 --device-type krbd --options noudev]
I1006 13:43:57.362632       1 nodeserver.go:414] ID: 48 Req-ID: 0001-0024-c7a90a82-4109-11ed-9a14-7b981f44cc3b-0000000000000002-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 rbd image: nomadstore/csi-vol-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 was successfully mapped at /dev/rbd0

Well, this was more informative but I’m not helped all that much by being shown what works. It’s what doesn’t work that would be helpful to see… It wasn’t entirely without usefulness though. The following commands allowed me to attempt things that ceph-csi seemingly never tried to do or did attempt but which failed silently:

rbd device list --format=json --device-type krbd
rbd --id admin -m manage01,manage02,manage03 --keyfile=***stripped*** map nomadstore/csi-vol-7403da80-fa1c-44e6-9794-2dca4ed3fbd7 --device-type krbd --options noudev

They failed inside ceph-csi-containers on worker03 but not on worker01 and worker02. They also failed outside ceph-csi on worker03 but worked outside ceph-csi on worker01 and worker02. So it wasn’t specifically a ceph-csi thing! As explained earlier, placing ceph-csi on one set of nodes and OSDs on another solved the issue and I’m not going to figure out why Ceph takes exception to node X asking to mount an RBD image by connecting to an OSD on node X.

Now, this was not nice to debug. You’d think that raising the verbosity level for the cephcsi-container from 5 to 9 or 99 would help:

      config {
        image = "quay.io/cephcsi/cephcsi:v3.7.1"

        args = [
          "--drivername=rbd.csi.ceph.com",
          "--v=5",
          "--type=rbd",
          "--nodeserver=true",
          "--nodeid=${node.unique.id}",
          "--instanceid=${NOMAD_ALLOC_ID}",
          "--endpoint=${CSI_ENDPOINT}",
          "--metricsport=${NOMAD_PORT_prometheus}",
          "--logtostderr=true",
        ]

Fun fact: no. Doesn’t do anything. Imagine if there hadn’t been commands logged so that I could run tests myself inside and outside the container to debug the issue! Nomad and Consul aren’t 100% when it comes to log message but they are way better than most container-related things. Credit also to Docker and Podman even though I’ve had to resort to debugging issue with them using strace at times. But things scheduled dynamically by something other than me is not easy to strace… So log messages aren’t just nice, they’re the only thing you’ve got to go on in many environments.

Anyway, now I have a PoC for this. I can demo that you can move workloads around with minimal downtime for maintenance or load-distribution purposes. But if a worker-node suddenly fails stateful workloads won’t failover. You have to explicitly tell Nomad that the allocation on the failed node is gone:

nomad volume status ceph-mariadb-test06
(read which allocation is using it)
nomad alloc stop d48619b7
nomad volume detach ceph-mariadb-test06 d48619b7

I’ll try to turn this into a workable Kubernetes alternative to keep my colleagues from jumping onto the K8s bandwagon. Because I don’t want to end up having to debug a failed Kubernetes cluster at 3 AM when I’m on call. Of course the time of day is less relevant when resolving the issue might take days due to poor logging and debug support. Look, if Kubernetes wants me to stop talking trash it needs to stop kicking me in the nuts!

Russia again

Who blew up the pipelines in the Baltic sea this week? Most people say Russia did it and I agree. Germany is trying to keep deliveries going, so they can be ruled out. Ukraine? Well, they have a pipeline going through their own territory. They can shut off Russian gas supplies to Europe by pushing a button. Blowing up two pipelines not in active use? Makes little sense.

But of course even the best theory – Russia did it – makes little sense. Russia too can just shut off gas supplies by pushing a button. Indeed they have in the case of Nordstream 1. They ain’t pumping nothing into that pipeline so blowing it up changes nothing. If anything Russia runs the risk of Germany thinking “Whoo-boy, better fix that pipeline to make gas deliveries possible.” If Russia just came right out and said “We won’t be supplying any gas through Nordstream 1 or 2″(the latter having never been taken into service, I know) Germany would have been in no doubt that they either had to make friends with Russia or get no gas.

But here’s the thing: Russia has consistently done stupid, pointless things that hurt Russia more than it hurts anyone else. So we can’t rule out that they did this stupid thing on the grounds that they wouldn’t do something stupid. They clearly do stupid things! It might seem like this doesn’t hurt Russia, since the pipelines were owned by Germany and were things that Germany was most interested in preserving. But having now attacked other countries’ energy infrastructure Russia has finally succeeded in getting the EU to deploy it’s military to protect such infrastructure. Something which should have been obvious to all involved nations some time ago.

It’s not that Russia can’t do things right. Annexing the occupied territory in Ukraine is – under the circumstances – a good move. It’s only a good move because they screwed themselves over for seven months straight but still, credit where credit is due. They can still screw up by claiming territory not currently held by Russian forces. That would make the whole referendum-ploy pointless and would bind Russia to capture territory that the Ukrainians are apparently very capable of defending. So we’ll see.

Also, why do people talk trash about the referendums as if though they are illegitimate because of the manner in which they were organized? The referendums could have been held with all the appropriate safeguards for freedom of speech, anonymous voting et cetera and they would still have been completely irrelevant. It’s not up to the population of Donetsk to decide if Donetsk is part of Ukraine. The people of Ukraine as a whole are the only ones who can redraw the borders of Ukraine. See the independence-referendum in Catalunia for a good comparison. They can ask for independence all they want. Unless a majority of Spain’s population votes in favor of their proposal: no change.

Now, I’m not entirely stupid… I understand that many of the people who want to undermine the recent referendums in Russian-held Ukrainian territory by pointing to procedural errors do so because they actually consider it to be entirely up to any small clique of people to declare independence, and have supported such referendums previously because they liked whoever was organizing them. So they are poorly placed to now suddenly acknowledge that maybe that’s a bad way of dividing up the world into tiny chunks. Belonging to some specific ethnicity doesn’t give you special powers to seize the territory of a sovereign nation.

This is anothing way in which Russia has screwed up, by now contradicting their many, many years of opposing the idea that ethnicity equals the privilege to seize arbitrary land. Now suddenly Russians can seize Ukrainian territory on ethnic grounds. A bad move for a country made up of so many ethnic groups. And something that their number one ally China is unlikely to appreciate.

The best route forward now would be for Ukraine to reject Russia’s annexation but announce their intention to resolve the issue of remaining Russian occupation of Ukrainian territory diplomatically. Russia won’t be able to complain about how Ukraine is invading “their” territory and might give Putin an opportunity to declare victory. The issue of what territory belongs to whom can then be resolved over time, with time working against Russia as it loses more and more of its trade with the industrialized world. Russia might in this situation reject any notions of peace because why not chop off even more of their own fingers? It’s like I’ve said: the only plan in Putin’s mind that could be served perfectly by everything that he has done is an attempt at trolling his successor and I suppose also the Russian people. If that’s his plan, he’s doing a marvelous job.

2022-09-30

Peskov further states that anyone talking about nuclear escalation behaves irresponsibly.

https://www.svt.se/nyheter/utrikes/direktrapport-forsamrat-sakerhetslage

Uhm… Yeeees. I would agree Mr. Peskov… But perhaps that’s a conversation to have with the Tsar of Russia? It seems a tad weird to make this statement to anyone else on planet Earth to be honest.

The Russian foreign minister also declares that even parts of the the four annexed parts of Ukraine not currently held by Russia should be viewed as annexed, citing de jure. Congratulations on winning Most audacious interpretation of international law-award of 2022! Invading a country, taking their land, redrawing their borders and then saying that legally the invaded country has to hand over even more land.

This undermines any claim Russia makes that the territory it occupies is Russian territory, an attack on which is an attack on Russia. With the same claim now being applied to Ukranian-held territory, one cannot support the idea of Russian-occupied Kherson being Russian without demanding that Ukraine hand over the rest of the region to Russia. And now Russia can just point to any piece of land, state it is Russian – Putin has previously argued that Ukraine is a natural part of Russia – and no-one can oppose it. That’s going to find a lot of traction around the world… Countries will be lining up to sign the “Russia can have anything it wants just by saying it”-petition. Well done on screwing up the one thing you could have done right!

Could they not stop delivering gas to Europe now please? It’s the biggest screw-up they could make! It’s just sitting there, ripe for the taking. How can Putin resist such a marvelous opportunity to screw Russia over? It will deprive Russia’s government of 1/3 of it’s annual revenue in a stroke. Europe has spent seven months finding replacements, powering up coal-plants and fast-tracking the Norway-Poland gas pipeline. How long will Russia last without gas-sales? I wish Europe had pulled the plug but that’s clearly not going to happen, so we can only put our hopes to the Russian penchant for screwing up.

2022-10-12

At the Moscow forum, Mr Putin described the leaks an “act of international terrorism”. But he said Russia was ready to resume supplies via the one intact part of Nord Stream 2.

https://www.bbc.com/news/world-europe-63231826

Oooohhh… So the point was to force Germany to open Nord Stream 2? But that’s idiotic… What value is there to forcing Germany to bring Nord Stream 2 online? I mean – first off – it’s not happening:

But Germany quickly rejected Mr Putin’s offer to send gas via Nord Stream 2.

https://www.bbc.com/news/world-europe-63231826

But even if it had worked, how would that help? You’re losing a war you started against a smaller country. Making a third party say “Fine, we’ll agree to do something we said we wouldn’t and use the other pipeline” doesn’t seem like it would even register on Russia’s list of priorities. It seems like it would rank somewhere above “Get North Korea to say Russia is fantastic” and just below “Find the missing remote to Putin’s favorite TV”.

Why not just stop delivering gas via Nord Stream 1 and then say “Gas will only be delivered via Nord Stream 2” if they think scoring points on this is worth even five minutes of their time? Blowing up pipelines seems a roundabout way to accomplish that task.

Over the past year, Russia cut its gas supplies to EU states by 88%, according to David Fyfe, chief economist with research firm Argus Media.

https://www.bbc.com/news/world-europe-63231826

Oh—key… Uhm… Then I guess we’re almost at where we need to be: Russia selling no gas to Europe. A bit surprising that they’ve gone that far. But appreciated none the less. I mean, the real goal is to deprive Russia of income but I doubt this 88% reduction in volume leaves them with 100% the income, even with higher gas prices. Interesting…

2022-10-14

The EU:s head of foreign affairs Josep Borrell warns Russia that its forces will be “annihilated” by the West if any form of nuclear weapons are used against Ukraine:

Any nuclear attack against Ukraine will trigger a retaliation, not a nuclear retaliation but such a powerful military retaliation that the Russian army will cease to exist.

https://www.svt.se/nyheter/eu-topp-rysslands-styrkor-kan-forintas (in Swedish)

Nicely done. I’ve been hoping for this clarification for months now. Russia using nukes is them demonstrating the weakness of their conventional forces. The response from NATO and the EU will be massive but there will be no need for a nuclear response. Western conventional weapon systems can easily cut off all Russian forces outside Russian territory and if they don’t surrender – annihilate them. Even implying that nuclear weapons are on the table when you’ve invaded a smaller country and have your ass handed to you is a sign of weakness that I don’t think Russia benefits from showing the world.

I guess it’s not that hard to cut off Russian forces within Russia either but that’s probably not on the table as an initial response. Russia uses a nuke => Russia and Belarus can just write off anything they have within Ukranian territory and any future access to Ukranian territory. If they try to play hardball at that stage => bad times will be had by all, Russia first and foremost. Not a good outcome. It’s much better for this to grind to a stalemate. Let the economic problems destabilize the regime in Belarus first. They were horribly poor before the war!

Then let the same forces grind down the Russian regime. That ought to take longer though. Belarus is a country that makes Russia look like some future space empire, on account of Russia having tractors. So Belarus first, Russia second. But we’ve seen their military fall apart way faster than expected so I wouldn’t bet on Russia holding out for too many years in the face of a huge shortfall in trade. Not that I’m a betting man, really… I’m more of a hedging man.

Thinking of Belarus and Russia, I’m reminded of this old meme:

No one has made a similar meme about Belarus because it would be too depressing.

2022-11-04

The UN official co-ordinating the grain deal, Amir Abdulla, praised Turkey’s role in the talks and welcomed Russia’s decision. However, German Foreign Minister Annalena Baerbock emphasised that Moscow had backed down.

“Russia has again tried to use hunger as a weapon, to use grain as a weapon,” she told German website Die Welt.

“The world community has made clear: No, we don’t believe your lies, we will continue to send ships… so the poorest in the world do not have to suffer so massively from this war of aggression.”

https://www.bbc.com/news/world-europe-63483133

Of for the love of… Like Russia can’t just sink them if they want? Don’t try to make it seem like Russia had no choice in this matter. It’s just that the choice consisted of
(1) stopping Ukrainian grain exports and declaring war on numerous other countries in the world(seizing or sinking a vessel from country X in international waters is either an act of piracy or an act of war against country X)
or
(2) not do something incredibly stupid and let the grain exports continue.

And for once Russia chose not to shoot itself in the foot. But it’s still a choice.

What I really object to is the remarkable lack of savoir faire. You don’t taunt your enemy when he gives in to your demands. See the text above about how Russia’s attempt to make Germany backpedal on Nordstream 2 would have been a complete waste of time and energy for Russia even if it had worked. It’s the same thing here. This kind of posturing is what I would expect from a street gang in LA who spray their graffiti over some other gang’s tags. Or from Russia… But I thought Germany had found some people with a modicum of sense these past few months. I stand corrected.

googler patch

googler (note the r at the end) is no longer being actively developed and it no longer works straight up. Turns out Google changed the class of some result entries. On line 2376

2375-                    # title_node, details_node, *_ = div_g.select_all('div.rc > div')
2376:                    title_node, details_node, *_ = div_g.select_all('div.tF2Cxc > div')
2377-                    if 'yuRUbf' not in title_node.classes:
2378-                        logger.debug('unexpected title node class(es): expected %r, got %r',
2379-                                     'yuRUbf', ' '.join(title_node.classes))

tF2Cxc has to be replaced by UK95Uc .