Design a site like this with WordPress.com
Get started

Bridge OS Kernel Panics On Macbook Pro

I’ve been getting “Your computer was restarted because of a problem” errors like this lately:

panic(cpu 0 caller 0xfffffff01e07db08): macOS watchdog detected
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 20P3045
macOS version: 22D68
Kernel version: Darwin Kernel Version 22.3.0: Thu Jan  5 20:17:53 PST 2023; root:xnu-8792.81.2~1/RELEASE_ARM64_T8010
KernelCache UUID: 66D3D9EAC3796DA60EBA5A1BCB588E81
Kernel UUID: A133ECF3-E2EF-3DEA-8A4C-D21463BB0B0D
Boot session UUID: 2D1ABF32-DA0B-4CF2-B82F-BF811BD8C2B3
iBoot version: iBoot-8419.80.7
secure boot?: YES
roots installed: 0
x86 EFI Boot State: 0x16
x86 System State: 0x0
x86 Power State: 0x0
x86 Shutdown Cause: 0x5
x86 Previous Power Transitions: 0x405060400
PCIeUp link state: 0x89271614
macOS kernel slide: 0x18600000
Paniclog version: 14
Kernel slide:      0x0000000017d2c000
Kernel text base:  0xfffffff01ed30000
mach_absolute_time: 0x7ce0fb8cd1
Epoch Time:        sec       usec
  Boot    : 0x640a4370 0x0003d4f9
  Sleep   : 0x640a7276 0x000d9fc2
  Wake    : 0x640ab429 0x000d7b2c
  Calendar: 0x640add0e 0x00007068

Zone info:
  Zone map: 0xffffffdc07cc0000 - 0xffffffe207cc0000
  . VM    : 0xffffffdc07cc0000 - 0xffffffdcee324000
  . RO    : 0xffffffdcee324000 - 0xffffffdd3aff0000
  . GEN0  : 0xffffffdd3aff0000 - 0xffffffde21654000
  . GEN1  : 0xffffffde21654000 - 0xffffffdf07cb8000
  . GEN2  : 0xffffffdf07cb8000 - 0xffffffdfee320000
  . GEN3  : 0xffffffdfee320000 - 0xffffffe0d4988000
  . DATA  : 0xffffffe0d4988000 - 0xffffffe207cc0000
  Metadata: 0xffffffdc00234000 - 0xffffffdc01a34000
  Bitmaps : 0xffffffdc01a34000 - 0xffffffdc01c68000

TPIDRx_ELy = {1: 0xffffffdf076d5088  0: 0x0000000000000000  0ro: 0x0000000000000000 }
CORE 0 is the one that panicked. Check the full backtrace for details.
CORE 1: PC=0xfffffff01ef4e848, LR=0xfffffff01ef4e848, FP=0xffffffeac9fdbf00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xffffffde216bd638: 0 pages, 223 threads: pid 0: kernel_task
Panicked thread: 0xffffffdf076d5088, backtrace: 0xffffffec820176c0, tid: 378
		  lr: 0xfffffff01ef1d44c  fp: 0xffffffec82017700
		  lr: 0xfffffff01ef1d25c  fp: 0xffffffec82017770
		  lr: 0xfffffff01f053220  fp: 0xffffffec820177e0
		  lr: 0xfffffff01f0521b8  fp: 0xffffffec820178a0
		  lr: 0xfffffff01eedd5fc  fp: 0xffffffec820178b0
		  lr: 0xfffffff01ef1ccd0  fp: 0xffffffec82017c60
		  lr: 0xfffffff01f5c6c1c  fp: 0xffffffec82017c80
		  lr: 0xfffffff01e07db08  fp: 0xffffffec82017cb0
		  lr: 0xfffffff01e065604  fp: 0xffffffec82017d10
		  lr: 0xfffffff01e06eac8  fp: 0xffffffec82017d60
		  lr: 0xfffffff01e067b58  fp: 0xffffffec82017e00
		  lr: 0xfffffff01e064b94  fp: 0xffffffec82017e70
		  lr: 0xfffffff01de81b40  fp: 0xffffffec82017ea0
		  lr: 0xfffffff01f51205c  fp: 0xffffffec82017ee0
		  lr: 0xfffffff01f5118b4  fp: 0xffffffec82017f20
		  lr: 0xfffffff01eee86c0  fp: 0x0000000000000000

That only gives you part of the story. The next step was to look at the Log Reports in Console for reports like panic-full-2023-03-10-073243.0003.ips.

{"roots_installed":0,"caused_by":"macos","macos_version":"Mac OS X 13.2.1 (22D68)","os_version":"Bridge OS 7.2 (20P3045)","macos_system_state":"running","incident_id":"2D1ABF32-DA0B-4CF2-B82F-BF811BD8C2B3","bridgeos_roots_installed":0,"bug_type":"210","timestamp":"2023-03-10 07:32:43.00 +0000"}
{
  "build" : "Bridge OS 7.2 (20P3045)",
  "product" : "iBridge2,3",
  "socId" : "0x00008012",
  "kernel" : "Darwin Kernel Version 22.3.0: Thu Jan  5 20:17:53 PST 2023; root:xnu-8792.81.2~1\/RELEASE_ARM64_T8010",
  "incident" : "2D1ABF32-DA0B-4CF2-B82F-BF811BD8C2B3",
  "crashReporterKey" : "c0dec0dec0dec0dec0dec0dec0dec0dec0de0001",
  "date" : "2023-03-10 07:32:43.90 +0000",
  "panicString" : <REMOVED FOR BREVITY>,
  "panicFlags" : "0x902",
  "bug_type" : "210",
  "otherString" : "\n** Stackshot Succeeded ** Bytes Traced 37852 (Uncompressed 115568) **\n",
  "macOSPanicFlags" : "0x0",
  "macOSPanicString" : "BAD MAGIC! (flag set in iBoot panic header), no macOS panic log available",
  "memoryStatus": <REMOVED FOR BREVITY>,
  "binaryImages": <REMOVED FOR BREVITY>,
  "processById": <REMOVED FOR BREVITY>

Note that Bridge OS is failing, indicating a Touch Bar issue. I found this SO post suggesting a cron script that restarts the touch bar every 3 minutes.

*/3 * * * * /usr/bin/pkill "Touch Bar agent" &>/dev/null; /usr/bin/killall "ControlStrip" &>/dev/null;

Edit your crontab with crontab -e then copy and paste that in. It’s not the cleanest solution, but if it gets the job done, I don’t really care.

I’m testing it now to see if it works…

Advertisement

How To Set Up Minikube Behind A VPN And Proxy On A Mac

I’ve been fighting with my local Minikube environment behind a VPN for months now. I resigned to using a remote server that doesn’t use the VPN, but decided to revisit the local environment setup recently and finally figured it out!

What I Think The Problem Was

I’ve always used the Virtualbox driver for Minikube. It works fine behind a proxy by just setting the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY variables. I think what messes up a lot of people is they forget to add all the Minikube-related IPs to their NO_PROXY variables. A cool thing about Kubernetes is it respects CIDR notation in your NO_PROXY. I even wrote a script to set up all the proxy variables.

But for some reason, Virtualbox doesn’t like my VPN…

$ minikube delete
🔥  Deleting "minikube" in virtualbox ...
💀  Removed all traces of the "minikube" cluster.
$ . proxy.sh http://proxy.example.com:8080
$ minikube start --driver virtualbox
😄  minikube v1.23.2 on Darwin 11.6.1
    ▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨  Using the virtualbox driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ HTTP_PROXY=http://proxy.example.com:8080
    ▪ HTTPS_PROXY=http://proxy.example.com:8080
    ▪ NO_PROXY=127.0.0.1,localhost,10.0.0.8/16,172.16.0.0/12,192.168.0.0/16,.example.com,.internal
    ▪ http_proxy=http://proxy.example.com:8080
    ▪ https_proxy=http://proxy.example.com:8080
    ▪ no_proxy=127.0.0.1,localhost,10.0.0.8/16,172.16.0.0/12,192.168.0.0/16,.example.com,.internal
❌  minikube is unable to connect to the VM: dial tcp 192.168.99.225:22: i/o timeout

        This is likely due to one of two reasons:

        - VPN or firewall interference
        - virtualbox network configuration issue

        Suggested workarounds:

        - Disable your local VPN or firewall software
        - Configure your local VPN or firewall to allow access to 192.168.99.225
        - Restart or reinstall virtualbox
        - Use an alternative --vm-driver
        - Use --force to override this connectivity check


❌  Exiting due to GUEST_PROVISION: Failed to validate network: dial tcp 192.168.99.225:22: i/o timeout

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

After reading the Minikube VPN documentation, I suspected my VPN client was configured to not allow local traffic, but when I tried to configure it, I couldn’t find any option to do so. So after reading the Pulse Secure VPN client documentation, I also suspected my IT department doesn’t allow users to configure the client. I tried the VMware driver with similar results. (The VMware driver seems to work now, and it works well, so maybe use that if you have VMware Fusion.) I broke up with Docker Desktop the first time because it didn’t respect NO_PROXY, and then again after they changed their license, and after all that, I just don’t want to use the Docker driver. So finally I checked out the HyperKit driver and that’s what solved all my problems. All I know about HyperKit is that it’s a Mac hypervisor developed by the Moby project, and it works with my VPN, so let’s get things set up…

Install Minikube and HyperKit

Install Homebrew if you haven’t already. Once you have Homebrew set up, install Minikube and HyperKit.

brew install minikube hyperkit

Create a Minikube Instance

Make sure you set up your proxy variables if required. Use my script if you want. The following command sets up a Minikube instance with a default configuration using HyperKit, but there’s a ton of options you can change. See minikube --help for those.

minikube start --driver hyperkit

There was an error about not being able to connect to k8s.gcr.io, but I just ignored it as it didn’t seem to affect the instance creation. Once the VM was created, I set up my Kubernetes/Docker clients. I wrote a script for that so you can just run . minikube.sh --driver hyperkit to set everything up. When I tried to pull an image, I got another error:

Error response from daemon: Get "https://registry-1.docker.io/v2/": proxyconnect tcp: dial tcp: lookup proxy.example.com on 172.16.129.1:53: read udp 172.16.129.14:44065->172.16.129.1:53: read: connection refused

It took me a while to resolve this, but when a co-worker posted a DNS config he used to get Docker running behind the VPN, I was able to adopt his method to configure Minikube’s DNS to use my host’s DNS server. Maybe Minikube has an option that does this automagically?

minikube ssh sudo resolvectl dns eth0 192.168.0.53
minikube ssh sudo resolvectl domain eth0 example.com

And then I was rocking!

    $ docker image pull timescale/timescaledb-postgis:2.3.0-pg13
    2.3.0-pg13: Pulling from timescale/timescaledb-postgis
    540db60ca938: Pull complete 
    a3cb73039552: Pull complete 
    39855706e49a: Pull complete 
    19d88c3ceadb: Downloading [===============================>                   ]  37.52MB/59.45MB
    9ef572e3c9bb: Download complete 
    261ea2d28080: Download complete 
    1716633ec467: Download complete 
    051e02f33f5f: Download complete 
    79ceec13a19e: Download complete 
    f43ce1b5bcdc: Download complete 
    9e276ca2472d: Download complete 
    3e6c8f42f360: Download complete 
    c9bfa14dc9b6: Download complete 
    101a245191b9: Downloading [===================>                               ]  16.91MB/44.11MB

Hope that works out for you with your VPN, but if not, let me know in the comments.

UPDATE: Things Are Still Broken

I thought my troubles were over, but it seems VMware and HyperKit drivers want me to use `minikube mount` to mount my host files in the VM so bind mounts will work. Virtualbox driver seem to do that automagically. I ran into this issue which required specifying the network IP address of the VM.

minikube mount /Users/me:/Users/me –ip 172.16.129.1

Then I ran into an error when running pytest:

OSError: [Errno 526] Unknown error 526

I followed some advice in this issue and increased msize to no avail.

minikube mount /Users/me:/Users/me –ip 172.16.129.1 –msize 524288

So now I’m back to just using a server without a VPN and Virtualbox. :*(