Microsoft 365

JFK gave a speech on the space program saying “we do these things, not because they are easy but because they are hard!” And I like that approach. That’s how you learn things. I’m going to throw myself into the madness that is Kubernetes again soon, having made Nomad and Consul do dependably what K8s thus far has failed to do. So, with that mountain climbed, let’s go back to the steeper one!

But these past two days I’ve moved four emails accounts from Microsoft 365 to an unaffiliated hosting provider. I got the customer’s admin password for Microsofts control panel. That didn’t help much because apparently adding some Microsoft App Validator-thing is now part of the onboarding process. I thought that would be that hard part…

I spent two hours trying to figure out how to get an IMAP client to connect to the accounts… Answer: you have to choose OAuth2 as an authentication method which – surprise, surprise – isn’t exactly default. Tools like IMAPsync for instance do not have them(yet). So why not generate an app password? Because you can only do that if you enable multi-factor authentication. But, the OAuth2-implemention that Microsoft uses is virtually MFA already, since the email client needs to connect to Microsofts webpage and do a little extra login-dance.

I got most of the emails over using the Microsoft Outlook email client. It took a long time but at least I didn’t have to supervise it by jumping around between Microsoft 365, Office 365, Exchange, Exchange Online Policy or Azure Directory during the process. I thought, after an hour and a half that Azure Directory was the answer to why IMAP with username and password wouldn’t work. Silly me…

I saw some interesting stuff in there while I was losing my mind one illogical jump from one control panel to another that is now decprecated so please visit this new one but oh-no we haven’t actually implemented all functionality in that one. No, wait… That’s cPanel API documentation I’m thinking of now.

Anyway, interesting stuff. Like some function called litigation hold. What a wonderfully American idea, to have a button to set your email account to “I’m being sued”. Also there are lots of policy things that you can tweak. If I haven’t made this clear yet: I wouldn’t want to use Microsoft 365 for my own applications. And the reason I wouldn’t want to do that isn’t that it’s expensive or unnecessary. I run a three physical server Ceph-cluster with redundant network switches in STP-mode for my own NAS – unnecessary is my middle name. No, I don’t want to use Microsoft 365 because it is too complex.

That’s why I brought up the whole Nomad-Consul-Ceph-Proxmox-STP-cluster-redundancy thing, because I can handle that! But Microsoft 365? It’s like staring into the abyss. You can spend hours just to figure out that the control panel you’ve been fiddling with doesn’t even do anything because a single switch in a completely different control panel is set to “Off”.

Now, if I for some reason had to manage a large corporation’s computer operations: I think I would choose Microsoft 365. I’m not actually sure it would be that secure or dependable. But I would have lots of buttons to easily indicate who should be able to do what. If someone asked “Can people send work-documents to their own personal gmail accounts?” I could pull up the Azure Directory-EOP-Mail Flow-Connector-rule thing that says “Nope, no work documents to outside domains” and everyone would be happy. It wouldn’t surprise me one bit if sending a zip-file containing work documents attached with a changed file-extension so that it didn’t look like a zip-file would circumvent that kind of stuff but no one would ask about that…

Anyway, we’re now going to tell customers who want to move from Microsoft 365 to our hosting that they are free to do so but that there is no practical way of getting their data over to us. Interesting how Microsoft have such great tools for moving data into their systems from IMAP-servers but won’t provide their customers with the same functionality if they should want to move data out. Where I work we have wacky systems that work the same whether customers are moving in or out. Silly.

The Ukranian situation

It’s going to be hard writing this without swearing but I’ll give it a shot.

Now, what the hell is Russia thinking? I don’t mean that rhetorically, I honestly want to know what they are thinking. Are they actually worried about NATO encroachment? They are… (counting) like 15 years too late there. What kind of chess-player does nothing until his opponent has captured the entire board except one or two squares?

Also, it was kind of a long-shot that any NATO country would move out offensive weapons from Eastern Europe but now that Russia has invaded a country in Eastern Europe, whatever possibilities there were for disarmament is gone. Let’s also consider how long it will take Sweden, Finland and Austria to apply for NATO membership. Clearly Russia takes whatever land NATO doesn’t protect. Genius piece of strategy this!

Maybe they never really cared about NATO encroachment, maybe that was just a cover-story? That would explain a few things, but what is the blinking crikey point of spinning such yarns for the West? To the Russian people the story now is something about protecting Russians in eastern Ukraine, not NATO threats. This choice of justification is a good thing because invading one of the few non-NATO countries in Eastern Europe on the grounds of worrying about NATO makes precious little sense.

Note how I don’t field arguments about rights and morality. We’re talking about sovereign nations. Morality is not on the table. But we can usually conjur up some logic that at least makes sense to the actor, even though we don’t necessarily think it’s very nice. Propping up the Belarussian government? Yeah, I suppose it’s nice to have on ally left in Eastern Europe. Not sure if it’s a solid long-term strategy though. Look at the former Warsaw pact and see how friendly the population is to Russia? Oh, they’re not? Maybe Russia imposing decades of dictatorship on them wasn’t a good way of building bridges? So how happy are the Belarussian people going to be when the regime finally falls?

This is kind of what we’re seeing in the Ukraine as well. Russian hasn’t exactly endeared themselves to the Ukranian people. Less so of course when annexing the Crimea! They really shot themselves in the foot with that. The Ukraine is never going to let Russia keep it’s presence in the Crimea. I see Russia having a naval base there now, sure. But fifteen years from now?

Maybe Russia intends to keep it by force? Then at least they aren’t stupid by way of disregarding the diplomatic relationship with the Ukraine and the opinions of their population. But then it’s stupid in another way, namely that they get themselves bogged down in a guerilla-war with the Ukraine for the Donetsk and Luhansk-regions and the Crimea. Russia covers like 12 time zones, why risk getting caught up in a Vietnam-style war for another few hundred square kilometers of territory that makes Soviet Russia look like an oppulent metropolis of wealth and prosperity?

No, this makes sense in exactly zero ways. Maybe I’m missing something, but it made way more sense when Saddam Hussein invaded Kuwait in 1990, which I think we all remember not working out great. But at least we could understand what made the move attractive to Iraq. Lots of oil, coast-line and a slightly iffy ability to cancel a lot of debt. No, it didn’t work out but the best case scenario made a lot of sense at least. What’s the best case scenario here? Massive economic growth by way of the Luhansk coal fields? Huge boost in tourism in Crimea? Detente with NATO after threatening to use nuclear weapons?

Maybe Putin has bought shares in some Russian weapons producers? That would at least make some sense… Is it immoral to throw Russia into war and sanctions to boost your personal stack of ill-gotten gains? Well, yeah! But that’s not on the table here. Reasoning about what actions of world leaders are moral or immoral is like arguing about “why” the laws of thermodynamics are the way they are. You can argue that until you’re blue in the face, but there’s no real answer. You bring a question(“why?”) into a domain where intent isn’t present. Similarly, condemning Putin for not being nice to the Ukraine is a perfectly valid thought-experiment about how the world should be but in reality it isn’t applicable. Nations are operated based on the crudest of base instincts. “What is best for my country?”

That’s why this annoys me, this isn’t even good for Russia! It’s like the whole Vietnam war or the long-term occupation of Afghanistan(pick any of them you want but I was thinking of the most recent one, not the Russian occupation 1979-1989 or the British one before that or the one before that. You get my point…) That doesn’t even help the west. Vietnam fell to communism and America absolutely trashed the general public’s trust in the government. Well done! And it’s not like it wasn’t obvious from the outset that it wasn’t going to work. It took JFK:s advisors two months tagging along with the South Vietnamese army to figure out why the US couldn’t win and also why the US couldn’t win. Stopping the spread of communism between North and South Vietnam? Nope. Stopping it on the border between Cambodia and Thailand? Easy as pie. The people of Thailand wouldn’t even consider going along with an idea that came from Cambodia, even if it were the most brilliant idea ever, just out of spite.

And I trust I don’t need to explain why occupying Afghanistan to make it into a modern democracy was a bad idea from day 1? The US didn’t even have the excuse of having forgotten how bad it is at those things. They tried it in Somalia and noped out before they so much as got a sun tan. Why would you 10 years later think Afghanistan would be any easier to set right? They have only ever succeeded in one thing, to oppose modernity. We mostly talk about their successful attempts at stopping modernity brought in by foreign powers but even during their long stints of autonomy they have a 100% success rate in staying in the Dark Ages.

So it’s not just Russia that’s shooting itself in the foot, but I fail to see how that helps. It’s not like the US debacles drove a bunch of countries to join the Warsaw pact, the way Russia is now acting as the number 1 salesman for NATO membership, however unwittingly. Not that Europe has done particularly well in this by making itself dependent on Russian natural gas. I guess it’s better than buying electricity produced by Russian nuclear power plants but that is also true of do-it-yourself kidney surgery… A blind man in Papua New Guinea saw this issue coming a mile away!

It’s not like we won’t get things back on track but I doubt a Putin-led Russia will be allowed to play a role in Europe. So we have to wait for him to retire first and hope that the next guy has some modicum of sense. I don’t expect Russia to be altruistic but not being self-destructive seems like a reasonable expectation. As noted earlier, the Ukraine will not let Russia hold the Crimea after this and the EU and NATO are probably not going to be willing to shake hands with Russia as long as the Ukraine cries foul. The EU and NATO don’t actually care about the Ukraine of course, but this is turning out to be a perfect fulcrum with which to lift Russia into a Sarlacc pit.

So new Russian leadership, Russia leaving Ukrainian territory and maybe stop propping up Belarussia? Then Russia can be brought back into the industrialized world. Which is how things are supposed to go. The developed world needs Russia to stop being obtuse.

By the way, what’s China going to do here? Because their participation in international politics is almost exclusively about making sure no one interfers with what China considers “internal matters”. Not sure how happy they are going to be that Russia now gives independence to separatists in a sovereign nation… They hate it when NATO does that(as do I, nationalism hasn’t served Europe super-well) so I don’t think “Russians in other countries will be supported in any struggle to take whatever land they happen to live on” is helping Sino-Russian relations. Then there’s also the whole “Russia invading a neighbor country”-thing which China – as a neighbor country – might not be super-cool with. What a bunch of God-damn brain surgeons…

But is this the end of the great European post-war peace, as NATO and EU spokesmen like to opine? There has been nothing of the sort! Soviet invasions of Hungary and Czechoslovakia, innumerable wars and ethnic cleansing in the Balkans and NATO bombing of Serbia over Kosovo spring to mind off the top of my head… Oh, then we have the Turkish invasion of Cyprus in the 70’s! Forgot about that one… No, this is nothing particularly new as such. It’s just weird to sabotage the trading relationships that Russia depends on for solvency in order to capture territory that NATO looks at and says “Nah, not interested.” Only to demonstrate that:

  • Russia doesn’t like the entire Warzaw pact joining NATO
  • Every country that joined NATO saved themselves from beind invaded by the Russian Federation
  • Any European country not already in NATO should join it quickly, or develop nuclear weapons
  • Russia is very keen on ethnic separatism

Now the first point is fair enough. No one was under the impression that Russia liked that all the countries that Russia held hostage by way of the Soviet union ended up joining NATO(save the Ukraine and Belarussia, countries that spent a large portion of the post-Soviet era being run by Russian puppets). So it is if anything a point that doesn’t need making. The other ones, well I think its pretty clear why the other three points harms the overall Russian agenda… NATO can fire their entire PR-department now. Russia has done their work for them.

And now they’re not even succeeding at the simplest part of the plan: capturing Ukraine. That was supposed to be a 72-hour thing! Now they might be getting desperate. How long before they use thermobaric bomb against Kyiv? That would even make China overtly denounce them for fear of ending up as collateral damage in marvelous the global sanctioning-craze that has seized the world. But it’s the kind of idiocy that we must now expect from them at this stage. Yeah, burn more bridges…

Update 2022-03-03:

The Russian president has told his French counterpart that Russia will successfully demilitarise Ukraine and render it neutral, which he said were his goals there.

BBC News report on a a call between France and Russia 2022-03-03

This is good news! Not that it is in any way a correct representation of what Russia actually had as its aims, but that’s hardly the point. We’re now seeing Russia start to set up a goal-post. If Russia had said it had set out to capture and occupy Ukraine then is could hardly save face when pulling out after X weeks. But by saying that the goal is to “demilitarise” Ukraine it will be much easier for Russia to pull out and claim victory.

If they choose their words carefully they might not even have to lie about achieving “demilitarisation”. Not that they are squeamish about bald-face lies, but if you can mix in some technically true statements in your lies that usually helps. Like if they claim to have “decimated Ukraine’s military hardware”. The word “Decimated” can be interpreted in many ways(“reduction by 10%” or “massive loss” are common uses of the word) and it is true that Ukraine has lost equipment. That will be replaced very shortly by the EU and NATO just as they are sending more and more guns to Ukraine at this very moment, but for a brief moment Russia will have achieved a reduction in Ukrainian military equipment.

This nonsense is much better than the (also nonsense) article published accidentally by a Russian media outlet a few days ago which lauded the Russian victory(ahem…) and their leadership’s decision to raise Russia up again to its former glory of a world superpower and sidelining the West. If Russia had publicly got behind that framing of the war then it would have had little choice but to continue fighting it until Ukraine was at least conquered, if not also occupied.

“Render Ukraine neutral” of course is… uhm… not going to happen, to put it mildly. It is right now the steel-tipped shoe kicking Russia in the shins with active support from the West. The EU and NATO are seizing the opportunity to test and demonstrate technology which hasn’t previously been used against the military forces of an industrialised nation. This will continue and Ukraine will probably be the most dedicated and outspoken opponent to Russia for the coming decades. It may in fact be that the EU needs to get Ukraine to chill a bit. Neutrality is the one thing we can carefully exclude as a possibility in Ukraine’s future.

I think we should be prepared for Russia stationing nuclear weapons in Belarussia in the near future however. It’s exceedingly cheap since the nukes already exist and moving them a few hundred kilometers west is easy enough. Based on the NATO response to the Russian invasion of Ukraine they will have to try very hard to make it look like they aren’t in a worse spot than they were before. It’s not like Russia will have a lot of spare cash to build up their military any time soon. It doesn’t really matter where the nukes are placed.

If anything Belarussia is a less strategically advantageous placement. Imagine in ten years if there is a coup in Belarussia and the new government invites Ukrainian forces in to suppress military units loyal to the old regime. Russian nukes could end up being seized in the process. Russia would certainly invade Belarussia to get them back at that point but it is usually the case that countries try to make their nukes as difficult to swipe as possible. Stationing some of them in the last European Warsaw-pact member still aligned to Russia should be seen as quite precarious. Russia will probably still do it just to make it seem like they are bravely countering NATO’s every move, even though it as a practical matter actually weakens them.

Update 2022-03-04:

President Putin has warned those opposing Russia’s actions in Ukraine “not to exacerbate the situation” by imposing more restrictions on his country.

BBC News feed 2022-03-04

I laughed long and hard at that one. Not going to well huh?

If he has any sense he will soon proclaim victory and an end to the “special military operation” which of course has gone better than planned. Based on these latest statements I think we can expect to hear demands from Russia to roll back sanctions within days. Some countries will agree, like China and India who think they can get away with it and are almost certainly correct that they can. But South Korea, Taiwan, North America in its entirety and Europe(barring Belarus of course) will say “Uhm, why don’t you come back to us when you’ve left Crimea and the Donbass?”

When Russia leaves the Crimean peninsula I think Ukraine will relinquish the Donbass and some sanctions will be lifted but until reparations are paid sanctions will persist. The last sanctions will probably only be lifted once Russia engages in some serious detente. Nuclear disarmament wouldn’t be bad but the US would have to follow suit. Free and fair elections would also make a signifcant difference I think. But of course just the “get Russia out of Crimea”-thing is going to take sooo long.

By the way, since the Russian invasion of Ukraine three former Warzaw-pact members have applied for EU membership: Ukraine, Georgia and Moldova. Ukraine is going to make it in once it reaches a deal with Russia on where the borders should be. Georgia is not since it is too far away. Georgia joining NATO might happen but not the EU. Moldova is so dirt-poor they had to introduce legislation to stop people selling their kidneys. So EU membership is not really on the cards for Moldova in my lifetime.

Also NATO announced Sweden’s and Finland’s participation in ongoing talks regarding the current situation. Countering NATO expansion seems to be going real well.

China, Cuba, or Venezuela can usually be relied upon to back Russia – this time they abstained.

BBC News feed 2022-03-04 about UN vote

Meanwhile in Russia the BBC, Deutsche Welle, Twitter and Facebook are blocked by the regime. I was about to propose that Facebook shut Russia out from their end but okey, if Russia wants to swing the axe that’s fine too. When’s the gas-tap going to be shut off and who will it be that makes the move? Customer or supplier?

2022-03-11

[China’s representative to the UN Zhang Jun] says it would encourage any country that has not yet destroyed their stockpiles of chemical weapons to do so as soon as possible.

BBC News feed 2022-03-11

I can sort of imagine how this played at the Kremlin:

– President Putin, the Chinese just gave a speech at the UN in our favor!
– Let me read that! … Uhm, so they said all countries with chemical weapons should destroy them as soon as possible?
– Yes, sir! Referring to Ukraine!
– But… We in Russia actually have chemical weapons. Ukraine doesn’t. That’s just a yarn we’ve spun to justify our invasion. So any call to destroy the world’s chemical weapons is going to have zero effect on Ukraine and a significant impact on Russia. I fail to see the silver lining here…
– Oh…

I’m not too impressed by the US refusal to furnish Ukraine with Mig jets provided by Poland because it makes the US look involved… The EU is sending tonnes of weapons with the expressed intention for them to be used against Russian forces. The world’s only real remaining superpower can probably afford to be seen acting a middle-man in the supply of jets from Poland to Ukraine. I’m not accusing them of being unsupportive, merely overly cautious. What’s Russia going to do? Invade Poland? They seem to have their hands full with Ukraine, I don’t think they’re going to fair much better against a member of NATO and the EU with… shall we say “a lengthy history of opposition to Russia” to be diplomatic? Russia might at most send a few cruise missiles at Polish air bases.

At that point of course it is pretty much open seasons on the Russian military. We might actually see a NATO or EU-enforced no-fly-zone over Ukraine and a blockade of Russian ports. I’m not saying that wouldn’t be an escalation, but it would be an escalation to Russia’s detriment. That an attack by Russia against a NATO member doesn’t not lead to a full-blown NATO retaliatory response but merely a “Russia is currently grounded. Aircraft or ships leaving Russia will be seized or destroyed.”-sort of tactic would signal that Russia isn’t taken all that seriously.

And Russia shouldn’t be seen as a particularly significant threat to the rest of the world. It’s down to one ally hostage in Europe – Belarus – a country that one day hopes it could reach the stratospheric economic success of East Germany… It’s one coup away from joining the “we who are sort of upset with Russia for it’s lengthy imposition of dictatorships on Eastern Europe”-group. If it weren’t for Russia still having nuclear weapons they would be a complete non-issue. But nuclear weapons aren’t credible weapons for anything other than a retaliation against a full-blown enemy invasion of your homeland. Russia trying rattle their nuclear saber if they don’t get to add Ukraine to their dwindling list of hostages implies that not even they think their conventional armed forces are very threatening…

2022-03-21

Russia’s Foreign Minister Sergei Lavrov has accused the US of restraining Kyiv from agreeing to Russian demands but did not appear to provide evidence

BBC News Feed

Right…

2022-03-24

While Biden spoke to the press Russia issued criticism against the western countries giving weapons to Ukraine. That support prolongs and intensifies the conflict in Ukraine, the Russian ministry of foreign affairs said in a statement.

SVT news article (in Swedish)

Well, every war would be short and peaceful if only one side had any weapons. The shortest possible war between Russia and Ukraine however would have been the one that Russia chose not to start. And that ship hasn’t just sailed – it has sailed, reached the East Indies and returned to port laden with spice!

Would Putin just please declare victory – the most complete and glorious victory throughout human history! – and pull back out. Stay in Crimea and Donbass of course! It will be the excuse for sanctions for many years to come.

If the West would just stop buying their god damn natural gas we the Russians would soon run out of money. With any luck the latest demand by Putin that natural gas from “hostile countries” be paid for in roubles will be denied by western countries, thus causing an end to the sale that way. Obviously that will lead to significant issues for Western Europe but if you make yourself dependent on Russia for critical energy supplies you gamble big. And then bad things happen. Countries like the Netherlands and Germany have benefited greatly from furnishing Russia with lots of money with which to build Tsar Putin’s new Russia in exchange for cheap energy and now they need to suffer a commensurate hardship.

Then maybe we can stop this nonsense of scaling down our own production of necessary fuels on the grounds that as long as nothing goes wrong we can typically get what we need.

2022-03-27

Would someone please get the dottering old man away from the microphone? I don’t disagree with Biden’s observation that Putin can’t be allowed to stay in power but being president in a democratic country doesn’t mean you get to dream up foreign policy like some freestyle rap battle. Not that the rest of the administration is that much better. What you do in these cases is double-down on calling for Putin’s removal, rather than let everyone know that the US president is a poorly controlled puppet. It’s like in that movie The Sum of all Fears where the Russian president claims to have ordered the use of chemical weapons – even though it was done by a general in violation of orders – because it’s better to be seen as a monster than someone who isn’t in control.

Zelensky also needs to do some more thinking before he talks. He is in no position to demand anything of NATO. He shouldn’t confuse the help he’s getting with anyone doing it out of some obligation or even with other countries caring about Ukraine. He gets precisely the equipment that helps Europe when it lands in Ukraine. Troops from EU countries in Ukraine or NATO-operated aircraft imposing a no-fly-zone? Nope, that doesn’t help Europe or NATO so none of that.

Maybe this has been explained to him because he now claims to be willing to discuss the status of the Donbass region. It’s not tenable to keep it as part of Ukraine no matter what Russia does. It’s full of Russians! Crimea though can’t be ceded or Ukraine loses lots of natural gas in the Black Sea. It was idiotic of Putin to annex it in the first place. It depends on water from a canal diverting water from a river in Ukraine. Want to guess how much water Ukraine has been letting through since 2014?

Now, I see no problem with leaving the issue of the Crimean peninsula entirely out a cease-fire agreement. Ukraine doesn’t need to use military force to retake Crimea, they just need to let sanctions brew for a while. So no point in forcing Russia to relinquish it now. Just don’t address the issue right now.

I wonder how the West will manage the sanctions at that point. Obviously the first sanctions to go will be the ones that are causing problems for Europe, like the oil embargo. But what about releasing the enormous mountains of foreign currency belonging to the Russian Central Bank? If the West refuses to release that money I wouldn’t be surprised if Russia announces their intention to re-open hostilities with Ukraine at which point Ukraine might also call for the money to be released. Tricky situation. I’d say “keep the money frozen”. “Europe and America doesn’t impose the sanctions on behalf of Ukraine and doesn’t remove them because a country threatens to violate international law.” would be a nice way of putting it.

For these reasons a cease-fire in Ukraine would be a very good period of time during which to bolster Ukraine’s armed forces. Russia is also going to regroup and rearm and a continuation of the war is far from impossible. More fighter jets, plenty of armored vehicles, artillery and well-trained infantry along with plenty of bunkers hidden in the vast plains. Ukraine did well this time around despite Europe reacting quite late to russian intentions to invade, probably thanks to Russia acting out the script to a Three Stooges movie and not a well-crafted military plan… But we shouldn’t assume Russia won’t correct their mistake.

Hopefully stage 2 of the war won’t be on the cards for at least a year. In that time sanctions can curtail what Russia considers to be feasible. As Germany moves away from russian gas it’s going to sting quite a lot at the Kremlin even if other sanctions are removed.

Nice of India to double their purchase of coal from Russia by the way. I find myself hoping for one of those riots the Twitterverse carries out whenever they hear someone with a conflicting opinion. Because India makes it quite clear that it chooses domestic expediency over the sovereignty of other nations. A boycott of Indian goods and services would be most appropriate. Then maybe India won’t need to buy so much coal from Russia because their factories will not have enough customers to warrant a significant use of elecitricity. A man can dream…

2022-03-31

Dang it. Starting to look like Russia isn’t going to follow through on their threat to cut off the gas-supply to Europe. Instead they will accept payment in euros and dollars as per usual and then buy roubles with that money themselves. Which they could have done all along… As it stands, nothing will put more pressure on Russia now than an end to gas exports, which is why it was a surprising development that they started talking tough about the currency used.

Because we see very clearly that central and eastern Europe isn’t going to stop buying Russian gas for a good couple of years, no matter what Russia does. Maybe if Russia attacked a NATO or EU country? Maybe then the flow of gas going west and the flow of money going east would stop? Maaaaybe… I can’t help but think of all the criticism lodged against those same countries against Switzerland for doing business with Germany during the second world war. Seems just vaguely hypocritical.

2022-04-02

Could the British and the Americans please stop telling China to not help Russia? Is that even a remote possibility? It’s like they’re trying to make China side with Russia… China doesn’t respond well to these kinds of threats. They’re more likely than not to do something that isn’t in their best interests just to show the world who’s the boss of China.

Praise China for their dedication to a peaceful resolution to the Ukraine crisis! Not because they’re actually helping of course… But China is unlikely to change course and support Russia just to villify themselves. If however they have to choose between looking weak and looking villainous, they’ll choose being a villain.

2022-04-05

The image from 19 March, first reported by the New York Times and confirmed by the BBC, directly contradicts Russian Foreign Minister Sergei Lavrov’s claim that footage of bodies in Bucha, that has emerged in recent days, was “staged” after the Russians withdrew.

https://www.bbc.com/news/60981238

Are you serious? If Russia had said “Terrible thing what happened to those Ukrainian civilians, typical Ukrainian nationalists murdering innocent people” I would have reserved judgement on who was behind it. Not so much because the Russian argument would have made any sense but the fog of war makes it very hard to know who did what and when. If Russia now claims that the dead civilians – in Russian occupied territories, who ended up in a mass grave with close-contact gunshots wounds – were not in fact actually murdered by anyone then even the most generous interpretation of events makes the Russian military look very guilty.

And again we find ourselves asking Why? Not some deep philosphical Why but a very pragmatic Why. Why kill civilians intentionally? Why try to dismiss accusations in a way that makes you look more guilty than if you had said “We must investigate these claims to ascertain if Russian forces were involved”? I.e. you look more guilty this way than if you openly entertained the notion that Russia was guilty of those acts.

Any plans to make the Russian military look unstoppable or even just competent failed pretty early on. Already by the second week people were starting to wonder what the hell the Russians were doing. But couldn’t they at least have maintained some air of… not being Nazi Einsatzgruppen filling up mass-graves in eastern Europe with civilians? Or are they going to use the same argument now as when they invaded Ukraine initially? “Well, the US invaded Iraq, so we can invade Ukraine! Also, the Nazis massacred civilians in Ukraine, and so can we.” Neither of which seems overly convincing when it comes to making the rest of Europe say “Oh, well then that’s all right! For a moment there we almost thought we had good reason to worry about our own safety”.

Russian sphere of influence

Is it just me or is Russia really bad at playing the game of international politics? Like, they want to keep NATO from placing military forces in countries that border on Russia. Cool beans! I see why they want that. But why then did they invade the Ukraine in 2014 and occupy the Crimean peninsula? Under what conceivable projections did they think invading a neighboring country would lead to anything other than a major influx of Russia-bordering countries to the NATO application procedure? Heck, there were talks about Georgia joining NATO for a while! And now they might be invading the Ukraine again, because they don’t like NATO expansion? It makes precious little sense.

Maybe this is just a side-show? Russia really isn’t threatened by NATO forces on their border. Or more accurately: NATO forces pose no greater threat to Russia on the Latvian-Russian border than they do at the French-German border. Europe is a very small place when war between industrialized nations is on the cards. It’s a shorter distance between Poland and the Belarussian-Russian border than the distance from the Belarussian-Russian border and Moscow. I’m arguing that these distances aren’t very relevant for ground-forces but of course jet-bombers and ICBMs barely notice those distances so they are even less relevant there.

So why kick up a fuss? They have a strong foothold in the Eastern Ukraine. Maybe they are under pressure? Why care about the Donbass? It’s not exactly full of gold mines and semiconductor factories… Coal and Soviet-era heavy industry is all. I thought Russia had enough of that already? Access to Crimea? No, that doesn’t track either, it doesn’t stretch far enough west. There’s always internal politics to keep in mind of course. Slavic unity, independence from the West… But people can’t eat those things and picking a fight with the West threatens trade relations that do support Russia economically.

So what are they up to?

Audible.com

At approximately 18:40 I thought I might acquire some kind of audiobook… thing. Maybe Audible.com? They have some subscription thing, right? At 19:35 I gave up trying to access Pandora’s Lab, part of the content included in my trial month, to have a nap.

Re-envigorated by my nap I was able to listen to the book I acquired using my monthly credit. Success! Does that mean I can access that Pandora’s Lab thing as well? Does it bollocks. I can buy it when I visit audible.com using Google Chrome – the same browser which let me listen to the aforementioned book, so I’m logged in it would seem – but listen to it as included content? Nope.

The Windows 10 App does show it as included, which seems like a step in the right direction:

3

This doesn’t work quite the entire way:

After reading up on this it turns out that their Windows 10 App doesn’t yet have support for Audible Plus content. Wow! So that’s where the Windows platform is at right now, getting the cold shoulder from content providers?

Well, okey then. I guess they only really support mobile apps? Nope, not Android at least. It also just let’s me buy that book.

So I’m coming away from this somewhat unclear on the benefits of Audible Plus. I’m not entirely confident in buying books from them either given this evening’s little expedition. Maybe I need to buy an iPhone, a Kindle or Alexa to have solid access to their content? I’ll admit that a Kindle would probably be okey but I wouldn’t be able to show my face at work carrying an iPhone. Techies don’t use iPhones! I’m not even sure I would be able to look myself in the mirror if I used an iPhone. Yet it would be better than an Alexa which in turn is only slightly better than being kidnapped by an Eastern European organ-harvesting syndicate.

What is it with providers of audio content? Because this is largely my experience with iTunes as well. I’ve set up a multi-master MariaDB cluster in the time it takes me to jump through their hoops of updates of the iTunes executable, my account details, managing which units can access me content et cetera. I haven’t played any of the content on my iTunes account for at least two years at this point.

Netflix? My mom watches lots of their content on her television and she’s not exactly a grade A hacker. Sure I had to set it up for her but it’s largely painless from that point. So for some reason video content is easy to access from many devices while audio content is provided by organisations that compete with one another in who can most closely resemble a Soviet era bureaucracy. “Here is your car. Please wait 8-10 years for your petrol purchasing-license to clear.”

Changing email provider – low downtime version

This is a method of switching email provider for a domain designed to minimize downtime. It was originally meant for IT people but not necessarily those who spend all week moving web hosting accounts. I realized thought that it was going to have to be a bit more complex than that. So there will be a weird mix of _inside baseball_ and stuff that only a noob would need to be told.

Let’s make up some addresses to use as placeholders and use my domain as an example of a domain to be moved.

Source provider(referred to as old provider)

IMAP serverimap.host-a.com
SMTP serversmtp.host-a.com
DNS servers:dns1.host-a.com, dns2.host-a.com

Destination provider(referred to as new provider)

IMAP servermail.xhost.com
SMTP servermail.xhost.com
DNS servers:ns1.xhost.com, dns2.xhost.com

Switching email provider

First we create two subdomains for deref.se called imap.deref.se and smtp.deref.se.

imap.deref.seCNAMEimap.host-a.com
smtp.deref.seCNAMEsmtp.host-a.com

The point of this is to move users with @deref.se-email addresses away from the old hosting providers names. We can’t keep using *.host-a.com for fetching emails once we’ve moved so sooner or later every device fetching and sending email using the deref.se domain name will have to be updated with a new IMAP server and a new SMTP server. By having these two records that lead back to the same place as the old records we can slowly move one user after another over to the new server names.

The standard approach is often to make a mad dash at some specific moment in time to switch all email-enabled devices from imap.host-a.com to mail.xhost.com so as to make the jump to the new hosting provider. But that’s error prone and stressful at best. Use server names decoupled from the source provider to make updating email clients workable.

Caveats so far

  • If you use SSL or TLS encryption at the old hosting provider you are going to want to issue certificates for these new subdomains( imap.deref.se and smtp.deref.se in this example ) before modifying any email clients. Otherwise big warnings and connection errors will affect email clients that use the new subdomains for fetching and sending email.
  • If you don’t know the password for all email accounts, set new ones when re-configuring the email clients. Sit down with Alice, set a new password for her email account and enter it into her phone and computer along with the new server names. Then go and do the same for Bob and so on.

Actual move

You should now have no email clients configured to connect to any host-a.com servers directly. They should all go via the new server names imap.deref.se and smtp.deref.se. You should also have a list of all the passwords used for the email accounts. If you use SSL/TLS you need to copy the SSL certificates used at the old hosting provider to the new one. You can test that they work correctly by changing your computer’s hosts file( https://www.howtogeek.com/howto/27350/beginner-geek-how-to-edit-your-hosts-file/ )

  1. Set the DNS-records imap.deref.se and smtp.deref.se to a TTL-value as close to 300 seconds as possible. Technically you could go lower but a five minute switch is typically fine. Do this at least X seconds before the actual move of the site where X is the old TTL-value.
  2. Create all the email accounts at the new hosting provider using the passwords from your list.
  3. Copy over the contents of each email account from the old provider to the new empty accounts over at the new provider. Consider using a tool like imapsync.
  4. Change the new records to the settings below:
imap.deref.seCNAMEmail.xhost.com
smtp.deref.seCNAMEmail.xhost.com

Congratulations, you have now moved email hosting to the new provider! I’d run imapsync once TTL seconds have passed. So five minutes after the DNS records were changed to point to the xhost.com-servers, running imapsync again will copy over any emails that happened to end up with host-a.com instead of xhost.com as the record-change propagated to DNS-servers across the internet.

Benefits so far

So here the big switch from one hosting provider to another is pretty much invisible to end users. They notice the preparations like re-configuring their email clients but that’s a brief visit from the local tech person, not a mad dash where dozens of people all try to change the settings on their email clients so that their email will work again.

Drawbacks so far

This approach requires more work for the person doing to move. Not creating provider-independent DNS records, leaving the TTL and just sending users a list of new settings and passwords takes less time. If you use the provider’s server names you can usually let them handle the SSL certificates.

On balance

How long is the CEO willing to have email work on his laptop but not his phone? Two weeks? One week? The closer you want to be to an imperceptible blip when switching email providers the more you should favor the benefits of this approach over the drawbacks.

Todo

  • Exim for all my nodes. Gotta keep emails from getting on the internet except for a few specific recipient addresses.
  • Kubernetes cluster. Maybe, just maybe I can learn to not hate it with such a burning passion.
  • Give Percona XtraDB Cluster another whack. Use the pcmk cluster to run a three-node setup and then use sysbench to stress it. Would be real nice to have Btrfs snapshots of the data dirs to test recoverability. If I could only find a way to recover from all three nodes disagreeing…
  • Test NixOS on a VM.

Done

  • Master-slave MySQL cluster on cluster1, cluster2, cluster3 to keep database load off Ceph. Pacemaker+Corosync automatically promotes and demotes nodes on failures, fencing if necessary. [ Still testing stability of setup. ]
  • WHM/cPanel testbed available via internet using separate VLAN.
  • Routing, traffic control and firewall lab. I wish VirtualBox VMs could have more network interfaces… Gonna run it on my Windows 10 workstation. [ Set up. Done with static and dynamic routing. ]

Scrapped

  • Setup Clonezilla server. [ Easier to just run Clonezilla from USB when needed. Didn’t particularly like the fact that Clonezilla has to be a DHCP server. ]

InfluxDB

Look at this graph of operation count per 10 seconds in my Ceph cluster and see if you can tell at which point I started installing CollectD on more hosts on my network to feed into InfluxDB.

Yeah…

The bump at the end is where I started moving some things around. After this time series there’s a 1 hour gap in InfluxDB data while I moved its storage from RDB to a local SSD. Also some reboots of the Proxmox nodes to put one network interface per node to a new VLAN.

Anyway, no high availability for my InfluxDB server but also a more reasonable load on Ceph. I’ll consider doing the same to logger(my Elastic Stack VM) because it’s also highly expendable and it’s easy to imagine it also causing a high number of write operations. It’s not like my Ceph cluster can’t handle this amount of writes but I just don’t like performance and log monitoring causing 80% of the load.

Home Lab – v1

I like high availability, probably more than I should. Banking, phone systems and the electrical grid do a good job with this but we have a lot of complex stuff at home nowadays which makes it trickier to keep them up and running reliably. It was more than ten years ago that I first tried to build a server setup that could keep things up and running even when individual servers failed or had to be brought down for maintenance. I didn’t have enough hardware with virtualization support so I had to use Xen(32-bit edition) with its paravirtualization support.

I used DRBD for shared storage and OCFS2 I think on top of that. It worked not so well. For various reasons I ended up having a single server with Solaris 11 and zRaid-5 storage so at least I had data redundancy even if the system as a whole wasn’t replicated. I later ended up with a master-slave setup with two identical Core i5-based “servers” where the master node replicated data over to the slave using Btrfs snapshots. The filesystems used Btrfs RAID1 on both nodes so there was a LOT of redundancy. There’s a story with zRaid-5, a weird hard drive failure and a couple of days of worrying behind this 2-server – each with internal 2-way replication – setup.

One of the servers in the master-slave setup gave up a few months ago so it was time to replace it with something new and now I think it’s pretty much complete.

Hardware

Node name: pve1
Microtower Atom C3558
16GB RAM
3.5″ hot swap spaces + 2 internal SATA connections
1 250 GB NVMe
1 250 GB SATA
1 500 GB SATA
1 2 TB 5400 RPM SATA
1 4 TB 5400 RPM SATA
4 Gbit LAN
IPMI

Node name: pve2
Microtower Xeon-D 1541
32 GB RAM
3.5″ hot swap spaces + 2 internal SATA connections
1 250 GB NVMe
2 250 GB SATA
1 500 GB SATA
1 2 TB 5400 RPM SATA
1 4 TB 5400 RPM SATA
Gbit LAN, 2 10GbaseT LAN
IPMI

Node name: pve3
Microtower Atom C3558
32GB RAM
3.5″ hot swap spaces + 2 internal SATA connections
1 250 GB NVMe
1 250 GB SATA
1 500 GB SATA
1 2 TB 5400 RPM SATA
1 4 TB 5400 RPM SATA
Gbit LAN
IPMI

Node name: nearline
Self-assembled Core i5-system
16GB RAM
Tandberg LTO3 tape drive
Hodgepodge of hard drives

Networking equipment:
Cisco RV082 Router
Netgear LB2120( for backup internet connection )
HPE 1920S switch

The tiny little screen to the top right is sort of a crude monitoring system. I don’t much feel like running a big LCD screen just to show the load on my servers. So this tiny screen tells me if any important hosts are down and what the load is on each Proxmox machine.

Structure

The nodes pve1-pve3 form a Proxmox 5.4 cluster with Ceph Luminous as shared storage. The journals for all Ceph OSDs is stored on NVMe partitions which took a while to set up since Proxmox’ own Ceph tools don’t want to do that. They say they do, but they don’t.

Some storage is kept out of Ceph because of reliability reasons. Basically I think of Ceph as a single point of (unlikely) failure. So virtual machines I want running even when I try to figure out why Ceph refuses to work are stored on LVM-thin volumes.

The nearline machine(not shown below since it is mostly turned off and so it makes little sense to monitor) is also a manifestation of my distrust in Ceph. I used to rsync data over to Btrfs-volumes once a day that I then snapshot. But the drives I put into the machine were junk so that had to stop. Now I got Bacula up and running again so therefor store backups on my cherished LTO-tapes. *hugs*

The Proxmox nodes use bonded network ports to connect to the HP Enterprise switch that serves mainly to connect the cluster together but it’s also the core switch of the network. The HP switch connects to my good old Cisco RV082 router which in turn connects to the fiber-modem that gives a nice 100 Mbit connection out. Now it also has 4G modem connected to WAN2 as a fallback.

The nodes with green links to the cloud symbol are stored on Ceph so can be migrated from physical hosts while running. Some nodes are not shown in the graphic above. Mostly they’re testbeds like my CloudLinux install with WHM and cPanel, a copy of my pacemaker-cluster and so on.

Software

Proxmox

A Debian-based virtualization platform with cluster and HA-support? Yes please… Has a great GUI and integration with Ceph. Kind of a pain to install new SSL certificates but it can be done. Basically Proxmox is an alternative to VMware vSphere and whatever Xen offers nowadays. Wish it had a way of configuring fencing for cluster nodes integrated with its own HA functionality. Has pretty good built-in performance monitoring as well.

Ceph

So I took the plunge to start using Ceph. It wasn’t entirely easy since I already had my cluster set up to use GlusterFS. It’s great to be able to move virtual machines from node to node using live migration but you can’t do that between separate shared storage systems now can you? I can handle some downtime but thought of it as an interesting experiment. Since I had DNS servers and MySQL servers set up in a cluster of virtual machines those virtual machines could be shut down one at a time, recreated on a new cluster with Ceph as a backend and then the process could be repeated one physical node at a time.

I didn’t need to create a new cluster but I figured I might as well go the entire way and upgrade to Proxmox 5.4. All in all there was like 5-10 minutes of down time inherent in the move and 1-2 hours of downtime because I’m a klutz who configures two servers to use the same IP address and then wonder why things don’t work so well.

Many decisions made about this setup reflect my lack of trust in Ceph but by now I’ve actually come to trust it quite well. It performed remarkably well when I screwed up the IP-addresses and other things. Haven’t encountered a split brain situation yet, which is more than I can say for GlusterFS(note: when increasing the node count in a GlusterFS setup, you have to change quorum levels manually…).

Ceph also has its own monitoring system which is nice.