Category: General Linux

PiHole weirdness

I kept seeing connections dropping out in Grafana and sometimes I even saw it in the browser with “No data” in all panels.

Things seemed to be in order with pdnsauth and pdnsrecursor so we had pihole as a suspect. Indeed keepalived didn’t think it was stable:

Sep 27 17:40:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:41:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:41:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:42:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:42:30 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:43:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:43:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:45:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:45:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:47:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:47:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:50:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:50:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:51:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:51:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:53:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:53:15 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:54:47 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:55:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:56:47 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:57:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 17:58:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 17:58:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:00:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:00:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:01:47 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:02:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:03:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:03:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:05:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:05:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:07:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:07:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:10:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:10:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:11:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:11:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:12:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:12:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:13:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:13:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:14:47 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:15:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:16:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:16:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:18:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:18:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:20:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:20:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:21:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:21:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:22:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:22:30 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Sep 27 18:23:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9
Sep 27 18:23:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0

Now pihole is out, we’re going straight to pdnsrecursor which checks certain names with pdnsauth, otherwise it’s the usual DNS resolution. Seems more stable:

googler patch

googler (note the r at the end) is no longer being actively developed and it no longer works straight up. Turns out Google changed the class of some result entries. On line 2376

2375-                    # title_node, details_node, *_ = div_g.select_all('div.rc > div')
2376:                    title_node, details_node, *_ = div_g.select_all('div.tF2Cxc > div')
2377-                    if 'yuRUbf' not in title_node.classes:
2378-                        logger.debug('unexpected title node class(es): expected %r, got %r',
2379-                                     'yuRUbf', ' '.join(title_node.classes))

tF2Cxc has to be replaced by UK95Uc .

Low power computers

So in a complete shock to everyone Europe is short on energy. Who could have guessed that depending on Russian gas, shutting down nuclear power plants and putting the absolute minimal effort possible into renewable energy would lead to energy shortages?! Bunch of god damn brain surgeons…

Well, maybe I don’t need the Xeon-D server running so that’s down. This means no Ceph which was kind of going to happen anyway. Interesting experiment. I’ve also switched from my 12-core AMD Ryzen to an Intel i3 NUC which struggles a bit to be honest. YouTube at 1080p leaves the CPU at 70% usage. At 720p it’s more like 55%.

2 cores and 2 threads per core, 16GB RAM and like 20GB of swap just in case. I use alacritty as a terminal emulator, btm for systems monitoring and of course Google Chrome as a web browser. Retroarch runs pretty well when the system isn’t too busy with other things. I’ve got Steam installed and presumably I’ll have to stick to 2D games like Shadowrun Returns, Huntdown, Guacamelee, Creeper World 3. I wonder how far I could run Factorio. That’s actually heavy purely at the simulation level. Steamworld Heist runs fine.

It will be interesting to see if Civ 4 will run. It’s a little bit of 3D and has to run with Proton/Wine so it’s not trivial. It should be noted that my screen is at 3440×1440 so that makes things a bit heavier. Assetto Corsa will of course have to run on the AMD Ryzen-workstation along with Subnautica and They Are Billions.

VLC runs just fine, I don’t have to run the snap version to avoid audio stuttering. I wonder how VS Code will run. I can run Python-development in Neovim but Node.js and Go wasn’t viable to develop like that.

Google Chrome isn’t easy on my CPU. YouTube at 720p60 and loading pages over at Hashicorp at the same time overpowers the system and makes YouTube downgrade to 480p. I’m using googler with w3m now and that works well. It’s really noticeable how resource intensive even simple static text pages are to load in a graphical browser, compared to a console-only browser with images disabled.

On Neovim as an IDE

I’m working on an ansible collection for setting up a workstation for myself. I’m using exa instead of ls, making bat available alongside cat, setting up neomvim etc. Neovim has some cool features for integrating with debugging tools but I have only made it works in a satisfactory way with Python and JavaScript. I’m probably going to remove JavaScript from the config and only use Neovim for Python development.

Because getting JavaScript in itself to work was an absolute pain. I thought that I – having done this already with Python – would be able to get JS and Go running more easily. No, JS was a pain and Go just fell apart. I’ll be using VS Code or JetBrains maybe for JavaScript, TypeScript and Go. This is no slight against Neovim. It’s amazing that you can make a terminal-based text editor work like an IDE at all but I’d argue that it needs to be assembled by the people making the integrations. It’s not viable to arrive at this kind of config by trial-and-error:

Plug 'mfussenegger/nvim-dap'
Plug 'nvim-telescope/telescope-dap.nvim'
Plug 'mxsdev/nvim-dap-vscode-js'

lua << EOF
local dap = require('dap')
dap.adapters.node2 = {
    type = "executable",
    name = "node-debug",
    command = "node",
    args= {"/home/cjp/.local/share/debuggers/vscode-js-debug/out/src/debugServerMain.js", "45635" }
}
dap.configurations.javascript = {
  {
    type = "pwa-node",
    request = "launch",
    name = "Launch file",
    program = "${file}",
    cwd = "${workspaceFolder}",
  },
  {
    type = "pwa-node",
    request = "attach",
    name = "Attach",
    processId = require'dap.utils'.pick_process,
    cwd = "${workspaceFolder}",
  }
}
EOF

lua << EOF
vim.g.markdown_fenced_languages = {
  "ts=typescript"
}
EOF

So I’ll be using Neovim instead of vim from now on and even use it to write and debug Python code, but I think we’ll have to wait for Neovim to be a generally viable IDE.

Addendum:

OK, I hate to say it but VS Code is pretty damn good. Credit where credit is due.

VSCode: Jinja templates huh? There are extensions for that. Should I show which ones?
Me: Uh… Sure. … Okey, and I just click to install? Well… That was easy.

And even on my Intel i3 NUC that coughs blood when I watch YouTube videos at 1080p it’s pretty damned responsive. I might get tired of having to use the mouse eventually but there’s a lot of polish to like in VS Code.

Sidenote:

This works great for finding examples of how certain things are done in Ansible that I can’t quite remember:

/ho/cj$ grep -C 10 -h -R "join(" ansible/
[mariadb]
log-bin                    = ON
server-id                  = {{ inventory_hostname_short[-1] }}
log-basename               = {{ inventory_hostname_short }}

{% if bootstrap %}
wsrep_cluster_address      = gcomm://
{% else %}
wsrep_cluster_address      = gcomm://{{ clusternodes|join(',') }}
{% endif %}

wsrep_cluster_name         = {{cluster_name}}
binlog-format              = ROW
default_storage_engine     = InnoDB
innodb_autoinc_lock_mode   = 2
wsrep_on                   = ON
wsrep_log_conflicts        = ON
wsrep_node_address         = {{ansible_facts["ens18"]["ipv4"]["address"]}}
wsrep_sst_receive_address  = {{ansible_facts["ens18"]["ipv4"]["address"]}}

Linux io troubleshooting

I was trying to find the source of some weird iowait on my main workstation and came across https://haydenjames.io/what-is-iowait-and-linux-performance/ which recommended the following

  • iostat – try it with the -xm 2 options for extended statistics, in megabytes and in two-second intervals.
  • iotop – top-like I/O monitor. Try it with the -oPa options to show the accumulated I/O of active processes only.
  • ps – use auxf, then under the “STAT” column “D” usually indicates disk iowait.
  • strace – view the actual operations issued by a process. Read the strace man page.
  • lsof – aft

Not bad. iostat -x is very wide so iostat -xsm 2 can be useful. It will show iowait unlike iostat -sm 2

iotop -oPa was great and showed that btrfs-transaction kernel thread was the big writer.

With apt install inotify-tools I got some nice inotify-based tools(as the name would suggest). Turns out that no files were actually written to the disks seeing iowait so I’ll demo it on another folder:

root@amd:~# inotifywatch -r /home/cjp/.cache/
Establishing watches...
Finished establishing watches, now collecting statistics.
^Ctotal  modify  close_write  open  create  delete  filename
53     33      5            5     5       5       /home/cjp/.cache/google-chrome/Default/Cache/Cache_Data/

It’s probably just another btrfs write-amplification-thing-a-majig. This showed the same thing:

lsof +D /path/to/btrfsmount

I’ve seen this before, just not with zero writes to the filesystem… Like 100KB=>500KB write I’ve seen. But 0=>500KB? That’s new.

Memo to self: be careful about putting Btrfs on SSDs.

Quick note on network debugging

Dropwatch to monitor packet drops in real time:

dnf install dropwatch
dropwatch –help
dropwatch -l kas
dropwatch> start

perf to figure out which software is making calls whose packets are dropped:

dnf install perf -y
perf record -g -a -e skb:kfree_skb
perf script

Log packets processed in iptables to dmesg:

iptables-legacy -t raw -A PREROUTING -p tcp –dport 9100 -j TRACE
dmesg
dmesg > dump.txt
ls -lh dump.txt
iptables-legacy -t raw -D PREROUTING -p tcp –dport 9100 -j TRACE

Still haven’t figured out why Kubernetes keeps dropping packets intermittently on one of three nodes(which one changes are workloads move around). It’s not conntrack being full or the pod receiving the traffic that’s dropping. It just enters ens18 and never enters the correct calico virtual interface so odds are the kernel drops ’em.

I can’t say I’m saddened by this turn of events. This is precisely the sort of stuff that I’ve been ranting about with this kind of “we handle it for you magically” stuff. Great when it works, not so great when you have to trace intermittent packet loss in a patchwork of vxlan and iptables entries managed by shadowy puppetmaster who doesn’t explain himself.

calico-node’s log with logscreenseverity set to debug and filelogging active:

2021-10-16 22:56:43.035 [INFO][8] startup.go 215: Using node name: kube02.svealiden.se
2021-10-16 22:56:43.196 [INFO][17] allocateip.go 144: Current address is still valid, do nothing currentAddr="10.1.173.128" type="vxlanTunnelAddress"
CALICO_NETWORKING_BACKEND is vxlan - no need to run a BGP daemon
Calico node started successfully

That was almost 4 hours ago…

Had a look at Nomad but I’m a little bit skeptical of that too. You seem to need Nomad, Consul and some networking thing to get a useful stack.

Oopsies

The tool for removing unused blocks in a filesystem is called fstrim. NOT blkdiscard. That tool discards ALL blocks. Don’t ask me how I came to discover this.

Memory hotplug problem in Proxmox

Edit: Oh, I’m in idiot. Proxmox clearly state that this doesn’t work in Ubuntu out of the box:

For Linux memory unplug, you need to have movable zone enabled, in the kernel config (not enabled by default on Debian/Ubuntu):

CONFIG_MOVABLE_NODE=YES

https://pve.proxmox.com/wiki/Hotplug_(qemu_disk,nic,cpu,memory)

Original post

Tried to reduce the memory used by a VM on ProxMox:

Parameter verification failed. (400)

memory: hotplug problem – 400 Parameter verification failed. dimm4: error unplug memory module

And in the syslog:


Jun 19 01:15:46 samba01 kernel: [6657625.965645] Offlined Pages 32768
Jun 19 01:15:46 samba01 kernel: [6657625.975153] Offlined Pages 32768
Jun 19 01:15:46 samba01 kernel: [6657625.977703] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:46 samba01 kernel: [6657625.977707] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:46 samba01 kernel: [6657625.977711] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:46 samba01 kernel: [6657625.977713] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:46 samba01 kernel: [6657625.977714] page dumped because: unmovable page
Jun 19 01:15:46 samba01 kernel: [6657625.982024] Offlined Pages 32768
Jun 19 01:15:46 samba01 kernel: [6657625.984611] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:46 samba01 kernel: [6657625.984615] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:46 samba01 kernel: [6657625.984619] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:46 samba01 kernel: [6657625.984622] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:46 samba01 kernel: [6657625.984623] page dumped because: unmovable page
Jun 19 01:15:46 samba01 kernel: [6657625.984647] memory memory50: Offline failed.
Jun 19 01:15:49 samba01 kernel: [6657628.969161] Offlined Pages 32768
Jun 19 01:15:49 samba01 kernel: [6657628.975996] Offlined Pages 32768
Jun 19 01:15:49 samba01 kernel: [6657628.977797] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:49 samba01 kernel: [6657628.977801] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:49 samba01 kernel: [6657628.977806] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:49 samba01 kernel: [6657628.977808] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:49 samba01 kernel: [6657628.977809] page dumped because: unmovable page
Jun 19 01:15:50 samba01 kernel: [6657628.982276] Offlined Pages 32768
Jun 19 01:15:50 samba01 kernel: [6657628.982783] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:50 samba01 kernel: [6657628.982786] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:50 samba01 kernel: [6657628.982790] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:50 samba01 kernel: [6657628.982792] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:50 samba01 kernel: [6657628.982793] page dumped because: unmovable page
Jun 19 01:15:50 samba01 kernel: [6657628.982815] memory memory50: Offline failed.
Jun 19 01:15:52 samba01 kernel: [6657631.979709] Offlined Pages 32768
Jun 19 01:15:53 samba01 kernel: [6657631.987515] Offlined Pages 32768
Jun 19 01:15:53 samba01 kernel: [6657631.987911] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:53 samba01 kernel: [6657631.987962] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:53 samba01 kernel: [6657631.987968] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:53 samba01 kernel: [6657631.987970] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:53 samba01 kernel: [6657631.987971] page dumped because: unmovable page
Jun 19 01:15:53 samba01 kernel: [6657631.991985] Offlined Pages 32768
Jun 19 01:15:53 samba01 kernel: [6657631.993757] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:53 samba01 kernel: [6657631.993761] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:53 samba01 kernel: [6657631.993765] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:53 samba01 kernel: [6657631.993768] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:53 samba01 kernel: [6657631.993769] page dumped because: unmovable page
Jun 19 01:15:53 samba01 kernel: [6657631.993792] memory memory50: Offline failed.
Jun 19 01:15:56 samba01 kernel: [6657634.990796] Offlined Pages 32768
Jun 19 01:15:56 samba01 kernel: [6657634.998121] Offlined Pages 32768
Jun 19 01:15:56 samba01 kernel: [6657634.998571] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:56 samba01 kernel: [6657634.998574] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:56 samba01 kernel: [6657634.998578] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:56 samba01 kernel: [6657634.998580] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:56 samba01 kernel: [6657634.998581] page dumped because: unmovable page
Jun 19 01:15:56 samba01 kernel: [6657635.002457] Offlined Pages 32768
Jun 19 01:15:56 samba01 kernel: [6657635.003626] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:56 samba01 kernel: [6657635.003631] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:56 samba01 kernel: [6657635.003635] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:56 samba01 kernel: [6657635.003637] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:56 samba01 kernel: [6657635.003638] page dumped because: unmovable page
Jun 19 01:15:56 samba01 kernel: [6657635.003661] memory memory50: Offline failed.
Jun 19 01:15:59 samba01 kernel: [6657637.999099] Offlined Pages 32768
Jun 19 01:15:59 samba01 kernel: [6657638.007138] Offlined Pages 32768
Jun 19 01:15:59 samba01 kernel: [6657638.007587] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:59 samba01 kernel: [6657638.007590] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:59 samba01 kernel: [6657638.007594] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:59 samba01 kernel: [6657638.007597] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:59 samba01 kernel: [6657638.007598] page dumped because: unmovable page
Jun 19 01:15:59 samba01 kernel: [6657638.012491] Offlined Pages 32768
Jun 19 01:15:59 samba01 kernel: [6657638.013243] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:15:59 samba01 kernel: [6657638.013246] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:15:59 samba01 kernel: [6657638.013254] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:15:59 samba01 kernel: [6657638.013256] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:15:59 samba01 kernel: [6657638.013257] page dumped because: unmovable page
Jun 19 01:15:59 samba01 kernel: [6657638.013291] memory memory50: Offline failed.
Jun 19 01:16:02 samba01 kernel: [6657641.011053] Offlined Pages 32768
Jun 19 01:16:02 samba01 kernel: [6657641.015726] Offlined Pages 32768
Jun 19 01:16:02 samba01 kernel: [6657641.016167] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:16:02 samba01 kernel: [6657641.016170] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:16:02 samba01 kernel: [6657641.016175] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:16:02 samba01 kernel: [6657641.016177] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:16:02 samba01 kernel: [6657641.016178] page dumped because: unmovable page
Jun 19 01:16:02 samba01 kernel: [6657641.020044] Offlined Pages 32768
Jun 19 01:16:02 samba01 kernel: [6657641.020495] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:16:02 samba01 kernel: [6657641.020499] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:16:02 samba01 kernel: [6657641.020503] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:16:02 samba01 kernel: [6657641.020505] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:16:02 samba01 kernel: [6657641.020506] page dumped because: unmovable page
Jun 19 01:16:02 samba01 kernel: [6657641.020529] memory memory50: Offline failed.
Jun 19 01:16:05 samba01 kernel: [6657644.027669] Offlined Pages 32768
Jun 19 01:16:05 samba01 kernel: [6657644.032558] Offlined Pages 32768
Jun 19 01:16:05 samba01 kernel: [6657644.033056] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:16:05 samba01 kernel: [6657644.033059] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:16:05 samba01 kernel: [6657644.033063] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:16:05 samba01 kernel: [6657644.033120] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:16:05 samba01 kernel: [6657644.033123] page dumped because: unmovable page
Jun 19 01:16:05 samba01 kernel: [6657644.037184] Offlined Pages 32768
Jun 19 01:16:05 samba01 kernel: [6657644.037639] page:ffffef54c64a1000 refcount:1 mapcount:0 mapping:ffff8c6fb1f8b880 index:0x0 compound_mapcount: 0
Jun 19 01:16:05 samba01 kernel: [6657644.037642] flags: 0x17ffffc0010200(slab|head)
Jun 19 01:16:05 samba01 kernel: [6657644.037647] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff8c6fb1f8b880
Jun 19 01:16:05 samba01 kernel: [6657644.037649] raw: 0000000000000000 00000000001d001d 00000001ffffffff 0000000000000000
Jun 19 01:16:05 samba01 kernel: [6657644.037650] page dumped because: unmovable page
Jun 19 01:16:05 samba01 kernel: [6657644.037673] memory memory50: Offline failed.

Tried to offline memory-chunks manually:

root@samba01:~# cat /sys/devices/system/memory/memory50/removable
1
root@samba01:~# echo offline > /sys/devices/system/memory/memory50/state
-bash: echo: write error: Device or resource busy
root@samba01:~# echo offline > /sys/devices/system/memory/memory50/state
-bash: echo: write error: Device or resource busy

Tried to force memory compaction:

root@samba01:~# echo 1 > /proc/sys/vm/compact_memory

But no. In the end I had to shut down the VM and restart it to make enforce the new memory limit. Not a big deal really but interesting. Just one more example of why we should set things up in such a way as to tolerate reboots. In this particular case it was my local network file share which is just fine to have down for two minutes but I do also have a read-only backup to take over when the primary is down.

Command line extravaganza

Some of us don’t particularly like using the computer mouse. It’s partly a matter of ergonomics – stretching out your arm to grab something wears thin after a couple of hours even if it’s just a few inches – and also a matter of keeping your fingers aligned on the keyboard so as to maintain touch typing. This has led to window managers like ratpoison which go in for completely removing the need for using a mouse.

I prefer i3 though it has a big problem with screen tearing so I only use it for non-graphical stuff. So video and games get to run in KDE and I have a separate VM where I do tech-stuff that’s heavy on text, terminals and so on. Some say the compositing… something-manager compton solves the issue but not for me.

We need more than just a keyboard-centric window manager to make it work. One of the most important missing pieces is browsing the web without a mouse. You could use something like w3m or lynx which is necessarily keyboard-centric as they are terminal-based. w3m can actually show images but that is obviously heresy. My solution of choice is Google Chrome with vimium. You can use Firefox with vimperator if you are so inclined.

In vimium you can press v once to start moving a caret around and another press of v starts marking a continous segment of text. Then it’s y to copy to clipboard. If you wonder why the characters vim appear so often in this context, it’s because the text editor vim is how hard core terminal junkies edit text files. It’s great but if you accidentally start vim you’ll feel like you’ve gone to crazy-town( :q and Enter will quit vim. You’re welcome). So maybe have a look at this crash course first: https://gist.github.com/dmsul/8bb08c686b70d5a68da0e2cb81cd857f

Terminal in itself

So what else do we need? Well, copy-pasting is the basis for all communication so it’s not sufficient to be able to copy paste from web browsers to text documents and terminals, but we must also be able to copy from text documents to terminals. Enter tmux! It creates virtual terminals in your terminal, sort of?

You get multiple screens and panes and can copy and paste within them, between them and also out to your system clipboard. Here’s the text copied from the screenshot:

root 5970 1 0 18:06 ? 00:00:00 /usr/lib/packagekit/pack
cjp 6017 4995 0 18:06 pts/0 00:00:00 tmux
cjp 6019 1318 0 18:06 ? 00:00:00 tmux
cjp 6020 6019 0 18:06 pts/1 00:00:00 -bash
cjp 6040 6019 0 18:06 pts/2 00:00:00 -bash
cjp 6052 6040 0 18:06 pts/2 00:00:00 ps -efd

My .tmux.conf file is pretty simple: .tmux.conf

fzf

This is a great fuzzy finder that helps you find text. Like a modern version of grep.

googler

Search Google via the terminal and open chosen result in a default or specified web browser.

ranger

A nice file browser in the terminal:

Ironical that I have to use my mouse to make screenshots of my completely keyboard-based VM.

bat

bat is like cat but for the modern era. The binary is called batcat on Ubuntu for some reason.

UEFI for Qemu VMs

That was a properly gatekeeping title. If you’re not in the know:
UEFI = Latest system for getting computers to start
VM = Virtual Machine
Qemu = Basis for Linux runs virtual machines using its own VM-speed-up-thing

So this is about how to boot virtual machines running on Qemu as if though the virtual machine had UEFI support. Which is a bit wonky… I was trying to set up Arch Linux on a Proxmox VM and it took a bit more time than initial projected.

You need to add a special EFI disk to a virtual machine if you want to be able to save UEFI entries but it maybe works sometimes. It might have been that I added and erased a bunch of entries while troubleshooting that broke it. Anyway, now I can add EFI entries and they show up just fine in efibootmgr-printouts but when I reboot only the default boot entries are available.

This turned out to give me a ground course in UEFI stuff, and make me go all little bit unhinged. Kind of like what happens when you try to understand how a company like Juicero could come to exist. Anyway, UEFI works on the principle of there being a EFI storage space on a connected hard drive. Note that this is not the same as the Proxmox EFI disk. Yeah, kind of complicated huh?

EFI needs to store lots of data about the operating systems that are available to boot on a computer. For Linux this is just a Linux kernel image and maybe an initramfs(which I needed because I thought I’d make my Arch VM use LVM for its root partition, because of course I did) but it can be other stuff as well. So the EFI disk you always need – even on a physical computer – is for operating system stuff and should be a few hundred megabytes in size.

The extra EFI disk you need to add for Proxmox is just a place to store things like “which Linux kernel to boot for Arch Linux”. That’s the part that seems to be kind of wonky. Anyway, the “real” EFI disk needs to be FAT32 formatted and throughout UEFI folders are separated from each other using backslashes like in Windows, not with forward slashes like in Linux. Not two things I choose when I have a choice.

All in all UEFI isn’t a bad step forward from BIOS as it replaces boot loaders in most aspects. That’s what I learned during all of this. Every UEFI implementation should have some fallback environment that it can start if nothing else works and there you can do basic command line stuff. Here’s how I got Arch Linux booting on my Proxmox VM finally:

I just realized I can’t copy-paste from the Proxmox UEFI shell… Well then. Okey, I guess I remember most of it and the script used to boot is available in the file system now.

Shell> FS0:
FS0>arch.nsh

I don’t know why it’s not “cd FS0” but it isn’t… Anyway, UEFI let’s you write and execute scripts, in this case arch.nsh:

\EFI\arch\vmlinuz-linux rw root=/dev/VG_root/root initrd=\EFI\arch\initramfs-linux.img

I made a proof of concept right in the UEFI shell:

Shell> FS0:
FS0>edit arch.nsh

There I could test different commands. Should it be \EFI\arch\vmlinuz-linux or EFI\arch\vmlinuz-linux(no leading backslash) ? Should it be initrd=\EFI\arch\initramfs-linux.img or initrd=”\EFI\arch\initramfs-linux.img” ? Using root=PARTUUID=LONGUUID was apparently not one of the right things but good old fashioned /dev/vgname/lvname worked fine.

I’m still not sure if scripts need to have the .nsh file ending but probably not. Anyway, now to figure out how to make Arch Linux autostart the network. And then another restart and booting using an EFI script!