Planet Raumfahrtagentur

April 26, 2020

lynxis

How to recover a supermicro board with a broken bios firmware

I was asked to take a look on a supermicro server which has been damaged by an firmware update. It was an Supermicro X10DRW-IT. The firmware update was tried by USB storage, but somehow failed.

After powering the system, it went like this:

  • On with all fans for 5 sec
  • Off for 5 sec
  • On with all fans for 5 sec
  • [..]

However it seems the IPMI is still booting through and won't be disconnected from the power as the remaining mainboard does, but the IPMI doesn't accept any bios update anymore.

As preparation I read up on the coreboot support for the Supermicro X10SLM+-F [0]

The bios chip is hidden under the raid controller if you've one.

Supermicro mainboard. Source: Supermicro X10DRW-i/X10DRW-iT USER’S MANUAL Revision 1.2b

First I've taken a look on the BIOS flash. To read the bios flash out, I've used a raspberry pi 3 with a SOIC-8 clip using the SPI bus.

Required tools:

  • raspberry pi with raspian (apt-get install flashrom)
  • some wire cables to the SOIC8 test clip
  • a SOIC-8 test clip (either the cheap ones from aliexpress or the expensive, high quality pomona 5250 [1]).

How you have to connect the SPI SOIC chip is described in [2]. Pin 1 of the SPI chip is where the small hole is on the.

Ensure you Disconnect both power supplies from the mainboard.

sudo flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=1000 -r bios
sudo flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=1000 -r bios2

sha256sum bios bios2
# ensure the checksum is equal, to ensure you read real things.

strings -n 16 bios
# try to get some strings out of it, ensure you not only read 0xffff or 0x0000.

Next I downloaded the supermicro bios update. Here you can find:

tree .

.
├── DOS
│   ├── AFUDOSU.SMC
│   ├── CHOICE.SMC
│   ├── FDT.smc
│   ├── FLASH.BAT
│   ├── Readme for X10 AMI BIOS-DOS+UEFI.txt
│   └── X10DRW9.B22
└── UEFI
    ├── Readme for X10 AMI BIOS-DOS+UEFI.txt
    ├── X10DRW9.B22
    ├── afuefi.smc
    ├── fdt.smc
    └── flash.nsh

ls -al X10DRW9.B22
-rw------- 1 lynxis users 16777216 Nov 22 16:07  X10DRW9.B22

Sound good, it's size is exact 16 MB, the same size as the bios flash. file also tell me what it is.

file X10DRW9.B22

X10DRW9.B22: Intel serial flash for PCH ROM

Great we found a firmware image with ifd (intel firmware descriptor). Now I've looked on the BIOS backup we read with the raspberry pi.

I used hexdump -C bios to see if the end contains a lot of 1s (or 0xffff in hex). Why? Because if you want to write a SPI flash, you can not just write to it like a hard drive. SPI flash chips are organised in blocks. A block is usally 64 kbyte. A single bit on a flash chip can only be written to a 0. If you want to write a single bit with a 1 where a 0 was before (0 -> 1), you've to erase the whole block, not only the address. An erase blocked is full of 1. To find out, if we have a half written flash, we can try to look on the end of the flash if there are a lot of 1s (or 0xffffffff).

hexdump -C is showing it quite nice

00c2ee20  4d 50 44 54 00 01 00 00  10 00 00 00 00 00 10 00  |MPDT............|
00c2ee30  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
01000000

This means, it only written up to 0x00c2ee30 (12.2 MB). Now we can look into the downloaded image, if it looks similiar. Maybe here starts configuration data. But no, it's missing some data here.

Next task is to flash the bios section. Bios section? The intel firmware description contains section. Similiar to a partition layout on a hard drive. On this platform there are 3 different sections

00000000:00000fff fd
00400000:00ffffff bios
00011000:003fffff me
  • fd stands for firmware descriptor
  • bios stands for the x86 firmware or UEFI
  • me for the management engine (also called on servers "Server Platform Services")

To flash only the partition you either have to use a recent flashrom version (at least 1.0) or you've to extract the layout file using the ifdtool (from coreboot). You can also use the last code snipped as layout.

flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=1000 -l layout -i bios -w X10DRW9.B22

It verifies it after writing to it. But still it doesn't work. My next thought was maybe the IPMI flash got damaged as well. The SPI flash of the IPMI is close by. So let's have a look. I was lucky to also have a SOIC-16 test clip available. I did the same procedure on the IPMI flash. However the flashrom in raspian was too old. The flash chip wasn't known to this version flashrom. I had to compile it myself.

sudo apt install git build-essential
git clone https://review.coreboot.org/flashrom.git
cd flashrom
make CONFIG_ENABLE_LIBPCI_PROGRAMMERS=no CONFIG_ENABLE_LIBUSB1_PROGRAMMERS=no
./flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=1000 -r ipmi

But it looks good so far. Running binwalk on it shows us

binwalk ipmi

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
103328        0x193A0         CRC32 polynomial table, little endian
1048576       0x100000        JFFS2 filesystem, little endian
4194304       0x400000        CramFS filesystem, little endian, size: 15216640, version 2, sorted_dirs, CRC 0xB1031FF3, edition 0, 8613 blocks, 1099 files
20971520      0x1400000       uImage header, header size: 64 bytes, header CRC: 0x3F1E0DA5, created: 2019-11-15 08:36:11, image size: 1537512 bytes, Data Address: 0x40008000, Entry Point: 0x40008000, data CRC: 0x310498CA, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: gzip, image name: "21400000"
20971584      0x1400040       gzip compressed data, maximum compression, has original file name: "linux.bin", from Unix, last modified: 2019-11-15 07:25:15
24117248      0x1700000       CramFS filesystem, little endian, size: 7458816, version 2, sorted_dirs, CRC 0xF5B9463B, edition 0, 3108 blocks, 466 files

Looks also good, however I want to be sure, it's the fine. I did first a backup, second overwritten with a file from the IPMI firmware update.

Still no change.

So what's wrong here? Is the power management controller damaged? The power supply are controller digital via I2C. Maybe it's somehow telling me something is wrong?

I was lucky, I didn't had a i2c sniffer around, otherwise I would have digged into it. I nearly gave it up, before I found out, that the backup file didn't worked with ifdtool. I exported the layout using the firmware update file, and not with the backup file. Usually firmware updates do not touch the ifd. It seems server boards are different. So the backup didn't contained an ifd. It wasn't only damaged in the end, also in the beginning. Not sure if this is a safety feature of the update. It might ensure at the beginning of an update the partial flash wouldn't be recognized as a working image. It's not a good thing booting a half working image.

I flashed the bios firmware update image and the board is back. To be sure, I flashed the ipmi backup on the SPI chip.

TLDR; So the fast way to recover a partial bios, do a backup first! Then flash the full image. At least for this generation it works.

Note: Depending on your specific hardware setup (cable length, test clip) you can increase or decrease the spispeed. spispeed=10000 => 10 MHz should be still ok. You'll notice the wrong spispeed if the reading or flashing fails.

[0] https://doc.coreboot.org/mainboard/supermicro/x10slm-f.html

[1] https://www.pomonaelectronics.com/products/test-clips/soic-clip-8-pin

[2] https://github.com/bibanon/Coreboot-ThinkPads/wiki/Hardware-Flashing-with-Raspberry-Pi

January 13, 2020

lynxis

My free software contribution in December 2019

Similiar to lolamby' regular post on his free software contribution, I want to start a similiar post.

The big chunk of the December I prepared together with the GSM team the celluar network at the 36c3 (36. Chaos Communication Congress). Every year we're building our own celluar network using free software project osmocom & open5gs. Osmocom is a community project around mobile communication. We used osmocom to run our core network (CN) of our 2G and 3G network. Open5gs is our LTE CN, which was interconnected to the osmocom CN. Here is a overview, every buble is an own daemon.

The 36c3 was a nice testing ground. We had to extend a couple of the services (e.g. osmomsc, osmogsup2dia, osmohnbgw). While the event we also like to enjoy the congress, so this is our excuse to not upstream our patches right a away. However we push our branches as it is to https://git.osmocom.org. Upstreaming will be more seen in January.

Joe the friendly coffee maker

Last week I visited thomasdotwtf from eventphone who has a jura coffee machine. We took one evening to look into it, how easy it is, to use a generic BLE device like a raspberry pi to control it. He has a Jura Z8 Automatic Coffee Machine which supports an IOS/Android app via Bluetooh LE.

Jura released (at least) two different apps to control it.

  • pl.mkssystems.juracoffee.household (Jura Coffee)
  • ch.toptronic.joe (J.O.E.)

Both apps supports ordering, changing properties of a coffee (e.g. how much water do you want to have or how much coffee should be in there?). mkssystems.pl seems to went out-of-service, but the internet archive still have an old version and they show a lot of coffee machine related products. As well as a small blue device [1].

This is the BlueFrog a bluetooth dongle to control Jura coffee machines.

What can you do with the Apps?

  • Configure your Coffee
  • Produce a Coffee
  • Statistics
  • Firmware updates

How we looked into?

  • bluetooth packettrace: We used the android btsnoop.log to retrieve a packet trace which we loaded into wireshark.
  • decompiled with different tools the .apk
  • loaded the source code into android studio

What we found out?

The J.O.E. application is using XML files to be configured for the different coffee machines. The XML defines products (e.g. a coffee, a green tea, ...), there are properties (e.g. how much coffee should be produced), statistics and settings. The article number defines the XML file to be used.

A firmware process including the update urls and the new firmware.

We tried to find the same commands which should work on the RS232/serial in the bluetooth packet trace, but there wasn't any. After looking further in the code, we found a lot UUIDs for characteristics including a human readable name. We discovered also an "encryption" method which uses 2x hardcoded keys as well an additional input of 8 bit from the BLE advertisment. The encryption look like a static key.

In the BLE advertisment, there are manufactoring data. In our case, the manufactoring data contains 27 bytes. If 16 bit will be used, it's little endian.

manufactoring data as hex (27 byte):
aa 05 06 03 d73a yyyy xxxx 5836 4435 01 c0 00 00 00 00 00 00 00 00 00 00 00

aa: key
05: BlueFrog Major Version
06: BlueFrog Minor Version
03: unused (maybe Patch Version?)
d73a: article number (the specific type of the machine)
yyyy: machine number
xxxx: serial number
5836: production date (Feb. 2017)
4435: production date UHCI (does UHCI means the bluefrog?) (Okt. 2016)
01: unused
c0: bitmask, define supported features

The production dates can be decoded and also validated using the application where it's shown in the connection fragment:

days: (i & 31)
month: ((i & 480) >> 5)
year: ((i & 65024) >> 9) + 1990;

What to do next?

Write a decrypt function which can parse pcap files and shows the message or write a dissector (lua) for wireshark with decryption function.

Find out how to map the XML files into commands towards the BlueFrog.

Bluetooth Interface

The good thing of BLE is, it's standarzied in the communication. BLE uses Bluetooth Attribute Protocol to communicate. The Bluetooth Attribute Protocol uses services and characteristics. A service is an object which can hold multiple characteristics. A characteristic can support one or more of the following operations read, write, notification, indication. Every service has a UUID as well a characteristic has a UUID. The Bluetooth Attribute Protocol has it's own methods to discover avaiable services and characterics. For more information please take a closer look into Bluetooth Low Energy.

As a general BLE device, the BlueFrog annouce itself on the BLE.

> hcitool lescan
LE Scan ...
C9:26:E8:4B:72:02 TT214H BlueFrog

> HCI Event: LE Meta Event (0x3e) plen 43                    #8 [hci0] 8.466202
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Scan response - SCAN_RSP (0x04)
        Address type: Random (0x01)
        Address: C9:26:E8:4B:72:02 (Static)
        Data length: 31
        Company: Ingenieur-Systemgruppe Zahn GmbH (171)
        Data: aa050603d73a080402005836443501c00000000000000000000000
        RSSI: -78 dBm (0xb2)

And further more we can also look for the services and characteristics via the gatttool.

> gatttool -b C9:26:E8:4B:72:02 --services -t random

attr handle = 0x0001, end grp handle = 0x0007 uuid: 00001800-0000-1000-8000-00805f9b34fb
attr handle = 0x0008, end grp handle = 0x0008 uuid: 00001801-0000-1000-8000-00805f9b34fb
attr handle = 0x0009, end grp handle = 0x0033 uuid: 5a401523-ab2e-2548-c435-08c300000710
attr handle = 0x0034, end grp handle = 0x003a uuid: 5a401623-ab2e-2548-c435-08c300000710
attr handle = 0x003b, end grp handle = 0xffff uuid: 00001530-1212-efde-1523-785feabcd123
> gatttool -b C9:26:E8:4B:72:02 --characteristics -t random

handle = 0x0002, char properties = 0x0a, char value handle = 0x0003, uuid = 00002a00-0000-1000-8000-00805f9b34fb
handle = 0x0004, char properties = 0x02, char value handle = 0x0005, uuid = 00002a01-0000-1000-8000-00805f9b34fb
handle = 0x0006, char properties = 0x02, char value handle = 0x0007, uuid = 00002a04-0000-1000-8000-00805f9b34fb
handle = 0x000a, char properties = 0x02, char value handle = 0x000b, uuid = 5a401524-ab2e-2548-c435-08c300000710
handle = 0x000d, char properties = 0x08, char value handle = 0x000e, uuid = 5a401525-ab2e-2548-c435-08c300000710
handle = 0x0010, char properties = 0x08, char value handle = 0x0011, uuid = 5a401529-ab2e-2548-c435-08c300000710
handle = 0x0013, char properties = 0x08, char value handle = 0x0014, uuid = 5a401528-ab2e-2548-c435-08c300000710
handle = 0x0016, char properties = 0x0a, char value handle = 0x0017, uuid = 5a401530-ab2e-2548-c435-08c300000710
handle = 0x0019, char properties = 0x02, char value handle = 0x001a, uuid = 5a401527-ab2e-2548-c435-08c300000710
handle = 0x001c, char properties = 0x02, char value handle = 0x001d, uuid = 5a401531-ab2e-2548-c435-08c300000710
handle = 0x001f, char properties = 0x0a, char value handle = 0x0020, uuid = 5a401532-ab2e-2548-c435-08c300000710
handle = 0x0022, char properties = 0x0a, char value handle = 0x0023, uuid = 5a401535-ab2e-2548-c435-08c300000710
handle = 0x0025, char properties = 0x0a, char value handle = 0x0026, uuid = 5a401533-ab2e-2548-c435-08c300000710
handle = 0x0028, char properties = 0x02, char value handle = 0x0029, uuid = 5a401534-ab2e-2548-c435-08c300000710
handle = 0x002b, char properties = 0x02, char value handle = 0x002c, uuid = 5a401536-ab2e-2548-c435-08c300000710
handle = 0x002e, char properties = 0x02, char value handle = 0x002f, uuid = 5a401537-ab2e-2548-c435-08c300000710
handle = 0x0031, char properties = 0x02, char value handle = 0x0032, uuid = 5a401538-ab2e-2548-c435-08c300000710
handle = 0x0035, char properties = 0x02, char value handle = 0x0036, uuid = 5a401624-ab2e-2548-c435-08c300000710
handle = 0x0038, char properties = 0x08, char value handle = 0x0039, uuid = 5a401625-ab2e-2548-c435-08c300000710
handle = 0x003c, char properties = 0x04, char value handle = 0x003d, uuid = 00001532-1212-efde-1523-785feabcd123
handle = 0x003e, char properties = 0x18, char value handle = 0x003f, uuid = 00001531-1212-efde-1523-785feabcd123

Bluetooth Services and Characteristics

service table
start end uuid name  
0x0001 0x0007 0x1800 Generic Access Profile  
0x0008 0x0008 0x1801 Generic Attribute Profile  
0x0009 0x0033 5a401523-ab2e-2548-c435-08c300000710    
0x0034 0x003a 5a401623-ab2e-2548-c435-08c300000710    
0X003b 0xffff 00001530-1212-efde-1523-785feabcd123    
Generic Access Profile (GAP) 0x0001 .. 0x0007
handle value handle properties uuid description
0x0002 0x0003 RW (0xa) 00002a00-0000-1000-8000-00805f9b34fb  
0x0004 0x0005 R (0x2) 00002a01-0000-1000-8000-00805f9b34fb  
0x0006 0x0007 R (0x2) 00002a04-0000-1000-8000-00805f9b34fb  
5a401523-ab2e-2548-c435-08c300000710 0x0009 .. 0x0033a
handle value handle properties uuid description
0x000a 0x000b R (0x2) 5a401524-ab2e-2548-c435-08c300000710 Machine Status
0x000d 0x000e W (0x8) 5a401525-ab2e-2548-c435-08c300000710 Product Start
0x0010 0x0011 W (0x8) 5a401529-ab2e-2548-c435-08c300000710 Service Control
0x0013 0x0014 W (0x8) 5a401528-ab2e-2548-c435-08c300000710 Update Product Progress
0x0016 0x0017 RW (0xa) 5a401530-ab2e-2548-c435-08c300000710 Product Progress
0x0019 0x001a R (0x2) 5a401527-ab2e-2548-c435-08c300000710 About
0x001c 0x001d R (0x2) 5a401531-ab2e-2548-c435-08c300000710  
0x001f 0x0020 RW (0xa) 5a401532-ab2e-2548-c435-08c300000710  
0x0022 0x0023 RW (0xa) 5a401535-ab2e-2548-c435-08c300000710  
0x0025 0x0026 RW (0xa) 5a401533-ab2e-2548-c435-08c300000710 Statistics command
0x0028 0x0029 R (0x2) 5a401534-ab2e-2548-c435-08c300000710 Statistics data
0x002b 0x002c R (0x2) 5a401536-ab2e-2548-c435-08c300000710  
0x002e 0x002f R (0x2) 5a401537-ab2e-2548-c435-08c300000710  
0x0031 0x0032 R (0x2) 5a401538-ab2e-2548-c435-08c300000710 Service Control Response
5a401623-ab2e-2548-c435-08c300000710 0x0034 .. 0x003a
handle value handle properties uuid description
0x0035 0x0036 R (0x2) 5a401624-ab2e-2548-c435-08c300000710  
0x0038 0x0039 W (0x8) 5a401625-ab2e-2548-c435-08c300000710  
Nordic DFU 00001530-1212-efde-1523-785feabcd123 0x003b .. 0xffff
handle value handle properties uuid description
0x003c 0x003d W- (0x4) 00001532-1212-efde-1523-785feabcd123 Nordic DFU_PACKET_CHARACTERISTI
0x003e 0x003f W N (0x18) 00001531-1212-efde-1523-785feabcd123 Nordic DFU_CONTROL_POINT_CHARACTERISTIC
  • R means read
  • W mean write with response
  • W- means write without response
  • N means notification

November 30, 2019