summer

I was asked to take a look on a supermicro server which has been damaged by an firmware update. It was an Supermicro X10DRW-IT. The firmware update was tried by USB storage, but somehow failed.

After powering the system, it went like this:

• On with all fans for 5 sec
• Off for 5 sec
• On with all fans for 5 sec
• [..]

However it seems the IPMI is still booting through and won't be disconnected from the power as the remaining mainboard does, but the IPMI doesn't accept any bios update anymore.

As preparation I read up on the coreboot support for the Supermicro X10SLM+-F [0]

The bios chip is hidden under the raid controller if you've one.

First I've taken a look on the BIOS flash. To read the bios flash out, I've used a raspberry pi 3 with a SOIC-8 clip using the SPI bus.

Required tools:

• raspberry pi with raspian (apt-get install flashrom)
• some wire cables to the SOIC8 test clip
• a SOIC-8 test clip (either the cheap ones from aliexpress or the expensive, high quality pomona 5250 [1]).

How you have to connect the SPI SOIC chip is described in [2]. Pin 1 of the SPI chip is where the small hole is on the.

Ensure you Disconnect both power supplies from the mainboard.

sudo flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=1000 -r bios
sudo flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=1000 -r bios2

sha256sum bios bios2
# ensure the checksum is equal, to ensure you read real things.

strings -n 16 bios
# try to get some strings out of it, ensure you not only read 0xffff or 0x0000.

Next I downloaded the supermicro bios update. Here you can find:

tree .

.
├── DOS
│   ├── AFUDOSU.SMC
│   ├── CHOICE.SMC
│   ├── FDT.smc
│   ├── FLASH.BAT
│   ├── Readme for X10 AMI BIOS-DOS+UEFI.txt
│   └── X10DRW9.B22
└── UEFI
    ├── Readme for X10 AMI BIOS-DOS+UEFI.txt
    ├── X10DRW9.B22
    ├── afuefi.smc
    ├── fdt.smc
    └── flash.nsh

ls -al X10DRW9.B22
-rw------- 1 lynxis users 16777216 Nov 22 16:07  X10DRW9.B22

Sound good, it's size is exact 16 MB, the same size as the bios flash. file also tell me what it is.

file X10DRW9.B22

X10DRW9.B22: Intel serial flash for PCH ROM

Great we found a firmware image with ifd (intel firmware descriptor). Now I've looked on the BIOS backup we read with the raspberry pi.

I used hexdump -C bios to see if the end contains a lot of 1s (or 0xffff in hex). Why? Because if you want to write a SPI flash, you can not just write to it like a hard drive. SPI flash chips are organised in blocks. A block is usally 64 kbyte. A single bit on a flash chip can only be written to a 0. If you want to write a single bit with a 1 where a 0 was before (0 -> 1), you've to erase the whole block, not only the address. An erase blocked is full of 1. To find out, if we have a half written flash, we can try to look on the end of the flash if there are a lot of 1s (or 0xffffffff).

hexdump -C is showing it quite nice

00c2ee20  4d 50 44 54 00 01 00 00  10 00 00 00 00 00 10 00  |MPDT............|
00c2ee30  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
01000000

This means, it only written up to 0x00c2ee30 (12.2 MB). Now we can look into the downloaded image, if it looks similiar. Maybe here starts configuration data. But no, it's missing some data here.

Next task is to flash the bios section. Bios section? The intel firmware description contains section. Similiar to a partition layout on a hard drive. On this platform there are 3 different sections

00000000:00000fff fd
00400000:00ffffff bios
00011000:003fffff me

• fd stands for firmware descriptor
• bios stands for the x86 firmware or UEFI
• me for the management engine (also called on servers "Server Platform Services")

To flash only the partition you either have to use a recent flashrom version (at least 1.0) or you've to extract the layout file using the ifdtool (from coreboot). You can also use the last code snipped as layout.

flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=1000 -l layout -i bios -w X10DRW9.B22

It verifies it after writing to it. But still it doesn't work. My next thought was maybe the IPMI flash got damaged as well. The SPI flash of the IPMI is close by. So let's have a look. I was lucky to also have a SOIC-16 test clip available. I did the same procedure on the IPMI flash. However the flashrom in raspian was too old. The flash chip wasn't known to this version flashrom. I had to compile it myself.

sudo apt install git build-essential
git clone https://review.coreboot.org/flashrom.git
cd flashrom
make CONFIG_ENABLE_LIBPCI_PROGRAMMERS=no CONFIG_ENABLE_LIBUSB1_PROGRAMMERS=no
./flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=1000 -r ipmi

But it looks good so far. Running binwalk on it shows us

binwalk ipmi

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
103328        0x193A0         CRC32 polynomial table, little endian
1048576       0x100000        JFFS2 filesystem, little endian
4194304       0x400000        CramFS filesystem, little endian, size: 15216640, version 2, sorted_dirs, CRC 0xB1031FF3, edition 0, 8613 blocks, 1099 files
20971520      0x1400000       uImage header, header size: 64 bytes, header CRC: 0x3F1E0DA5, created: 2019-11-15 08:36:11, image size: 1537512 bytes, Data Address: 0x40008000, Entry Point: 0x40008000, data CRC: 0x310498CA, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: gzip, image name: "21400000"
20971584      0x1400040       gzip compressed data, maximum compression, has original file name: "linux.bin", from Unix, last modified: 2019-11-15 07:25:15
24117248      0x1700000       CramFS filesystem, little endian, size: 7458816, version 2, sorted_dirs, CRC 0xF5B9463B, edition 0, 3108 blocks, 466 files

Looks also good, however I want to be sure, it's the fine. I did first a backup, second overwritten with a file from the IPMI firmware update.

Still no change.

So what's wrong here? Is the power management controller damaged? The power supply are controller digital via I2C. Maybe it's somehow telling me something is wrong?

I was lucky, I didn't had a i2c sniffer around, otherwise I would have digged into it. I nearly gave it up, before I found out, that the backup file didn't worked with ifdtool. I exported the layout using the firmware update file, and not with the backup file. Usually firmware updates do not touch the ifd. It seems server boards are different. So the backup didn't contained an ifd. It wasn't only damaged in the end, also in the beginning. Not sure if this is a safety feature of the update. It might ensure at the beginning of an update the partial flash wouldn't be recognized as a working image. It's not a good thing booting a half working image.

I flashed the bios firmware update image and the board is back. To be sure, I flashed the ipmi backup on the SPI chip.

TLDR; So the fast way to recover a partial bios, do a backup first! Then flash the full image. At least for this generation it works.

Note: Depending on your specific hardware setup (cable length, test clip) you can increase or decrease the spispeed. spispeed=10000 => 10 MHz should be still ok. You'll notice the wrong spispeed if the reading or flashing fails.

[0] https://doc.coreboot.org/mainboard/supermicro/x10slm-f.html

[1] https://www.pomonaelectronics.com/products/test-clips/soic-clip-8-pin

[2] https://github.com/bibanon/Coreboot-ThinkPads/wiki/Hardware-Flashing-with-Raspberry-Pi

[Reposted from lokrund2015]

scnr..

Similiar to lolamby' regular post on his free software contribution, I want to start a similiar post.

The big chunk of the December I prepared together with the GSM team the celluar network at the 36c3 (36. Chaos Communication Congress). Every year we're building our own celluar network using free software project osmocom & open5gs. Osmocom is a community project around mobile communication. We used osmocom to run our core network (CN) of our 2G and 3G network. Open5gs is our LTE CN, which was interconnected to the osmocom CN. Here is a overview, every buble is an own daemon.

The 36c3 was a nice testing ground. We had to extend a couple of the services (e.g. osmomsc, osmogsup2dia, osmohnbgw). While the event we also like to enjoy the congress, so this is our excuse to not upstream our patches right a away. However we push our branches as it is to https://git.osmocom.org. Upstreaming will be more seen in January.

Last week I visited thomasdotwtf from eventphone who has a jura coffee machine. We took one evening to look into it, how easy it is, to use a generic BLE device like a raspberry pi to control it. He has a Jura Z8 Automatic Coffee Machine which supports an IOS/Android app via Bluetooh LE.

Jura released (at least) two different apps to control it.

• pl.mkssystems.juracoffee.household (Jura Coffee)
• ch.toptronic.joe (J.O.E.)

Both apps supports ordering, changing properties of a coffee (e.g. how much water do you want to have or how much coffee should be in there?). mkssystems.pl seems to went out-of-service, but the internet archive still have an old version and they show a lot of coffee machine related products. As well as a small blue device [1].

This is the BlueFrog a bluetooth dongle to control Jura coffee machines.

manufactoring data as hex (27 byte):
aa 05 06 03 d73a yyyy xxxx 5836 4435 01 c0 00 00 00 00 00 00 00 00 00 00 00

aa: key
05: BlueFrog Major Version
06: BlueFrog Minor Version
03: unused (maybe Patch Version?)
d73a: article number (the specific type of the machine)
yyyy: machine number
xxxx: serial number
5836: production date (Feb. 2017)
4435: production date UHCI (does UHCI means the bluefrog?) (Okt. 2016)
01: unused
c0: bitmask, define supported features

days: (i & 31)
month: ((i & 480) >> 5)
year: ((i & 65024) >> 9) + 1990;

> hcitool lescan
LE Scan ...
C9:26:E8:4B:72:02 TT214H BlueFrog

> HCI Event: LE Meta Event (0x3e) plen 43                    #8 [hci0] 8.466202
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Scan response - SCAN_RSP (0x04)
        Address type: Random (0x01)
        Address: C9:26:E8:4B:72:02 (Static)
        Data length: 31
        Company: Ingenieur-Systemgruppe Zahn GmbH (171)
        Data: aa050603d73a080402005836443501c00000000000000000000000
        RSSI: -78 dBm (0xb2)

> gatttool -b C9:26:E8:4B:72:02 --services -t random

attr handle = 0x0001, end grp handle = 0x0007 uuid: 00001800-0000-1000-8000-00805f9b34fb
attr handle = 0x0008, end grp handle = 0x0008 uuid: 00001801-0000-1000-8000-00805f9b34fb
attr handle = 0x0009, end grp handle = 0x0033 uuid: 5a401523-ab2e-2548-c435-08c300000710
attr handle = 0x0034, end grp handle = 0x003a uuid: 5a401623-ab2e-2548-c435-08c300000710
attr handle = 0x003b, end grp handle = 0xffff uuid: 00001530-1212-efde-1523-785feabcd123

> gatttool -b C9:26:E8:4B:72:02 --characteristics -t random

handle = 0x0002, char properties = 0x0a, char value handle = 0x0003, uuid = 00002a00-0000-1000-8000-00805f9b34fb
handle = 0x0004, char properties = 0x02, char value handle = 0x0005, uuid = 00002a01-0000-1000-8000-00805f9b34fb
handle = 0x0006, char properties = 0x02, char value handle = 0x0007, uuid = 00002a04-0000-1000-8000-00805f9b34fb
handle = 0x000a, char properties = 0x02, char value handle = 0x000b, uuid = 5a401524-ab2e-2548-c435-08c300000710
handle = 0x000d, char properties = 0x08, char value handle = 0x000e, uuid = 5a401525-ab2e-2548-c435-08c300000710
handle = 0x0010, char properties = 0x08, char value handle = 0x0011, uuid = 5a401529-ab2e-2548-c435-08c300000710
handle = 0x0013, char properties = 0x08, char value handle = 0x0014, uuid = 5a401528-ab2e-2548-c435-08c300000710
handle = 0x0016, char properties = 0x0a, char value handle = 0x0017, uuid = 5a401530-ab2e-2548-c435-08c300000710
handle = 0x0019, char properties = 0x02, char value handle = 0x001a, uuid = 5a401527-ab2e-2548-c435-08c300000710
handle = 0x001c, char properties = 0x02, char value handle = 0x001d, uuid = 5a401531-ab2e-2548-c435-08c300000710
handle = 0x001f, char properties = 0x0a, char value handle = 0x0020, uuid = 5a401532-ab2e-2548-c435-08c300000710
handle = 0x0022, char properties = 0x0a, char value handle = 0x0023, uuid = 5a401535-ab2e-2548-c435-08c300000710
handle = 0x0025, char properties = 0x0a, char value handle = 0x0026, uuid = 5a401533-ab2e-2548-c435-08c300000710
handle = 0x0028, char properties = 0x02, char value handle = 0x0029, uuid = 5a401534-ab2e-2548-c435-08c300000710
handle = 0x002b, char properties = 0x02, char value handle = 0x002c, uuid = 5a401536-ab2e-2548-c435-08c300000710
handle = 0x002e, char properties = 0x02, char value handle = 0x002f, uuid = 5a401537-ab2e-2548-c435-08c300000710
handle = 0x0031, char properties = 0x02, char value handle = 0x0032, uuid = 5a401538-ab2e-2548-c435-08c300000710
handle = 0x0035, char properties = 0x02, char value handle = 0x0036, uuid = 5a401624-ab2e-2548-c435-08c300000710
handle = 0x0038, char properties = 0x08, char value handle = 0x0039, uuid = 5a401625-ab2e-2548-c435-08c300000710
handle = 0x003c, char properties = 0x04, char value handle = 0x003d, uuid = 00001532-1212-efde-1523-785feabcd123
handle = 0x003e, char properties = 0x18, char value handle = 0x003f, uuid = 00001531-1212-efde-1523-785feabcd123

Lesen auf dem Bildschirm bedeutet immer ein Lesen unter erschwerten Bedingungen.