An unlikely PowerPC desktop

[jn926]

A while ago, someone offered me 40 WiFi access points and a few switches, that were previously installed in a student dormitory. I like free hardware, so I accepted.

Note: This post is split up because I misunderstood how phpBB works. I hope it's still readable.

stash.jpg (159.66 KiB) Viewed 8132 times

Quick facts:

• Product: LANCOM L-321agn Wireless (R2)
• SoC: Freescale/NXP MPC8314E (32-bit PowerPC e300)
• RAM: 128 MiB
• Flash: 256 MiB parallel NAND flash
• Ethernet: 1 Gbit/s
• WLAN: Atheros AR8035-A (ath9k) mini-PCIe card
• a second unpopulated mini-PCIe slot
• Power supply: 12V barrel jack
• Serial port: LANCOM 8-pin DIN connector

inside.jpg (204.18 KiB) Viewed 8132 times

My first impression was that it's rather clean-looking board. LANCOM builds nice hardware.

But I'm curios, so I wanted to run my own code on it.

We (a few friends at CCCAC, and I) tool a look at the official update files that can be downloaded from the LANCOM website but didn't fully understand the format. They contain PowerPC code and some compressed data, but there were gaps in the LZMA stream. Turns out, this "UPX" (not to be confused with the other UPX) is a LANCOM-specific format, so I wrote some tools to parse/generate it.

One thing to know about LANCOM is that they've avoided using the usual open source stack of U-Boot + Linux + OpenWRT for as long as possible. This device runs proprietary first- and second stage bootloaders, and a proprietary operating system called LCOS.


Hands on

After buying the right kind of serial port adapter cable (12€ on eBay), I watched it boot. This unit had inexplicably erased its firmware, so it offered "Xmodem Upload":

Code: Select all

WDG:
@C@ 
memory test:  1 2 3 4 5  OK

Found primary and mirror bbt

NO VALID FIRMWARE!

# FLASHROM-Upload
| LANCOM L-321agn Wireless (R2)
| Copyright (C) LANCOM Systems
| Ver. 4.00.0001 / 12052014 / 132211

Start Xmodem Upload
...

Receive Error

Upload aborted

Without seeing the bootloader's code, I wasn't sure what kind of file it expected; but the bootloader was in flash. We (a few friends at CCCAC, and I) desoldered the flash and tried to read it with the Glasgow Interface Explorer, which supports parallel NAND flash (following the "ONFI" standard), but it didn't work reliably.

[jn926]

jtag.jpg (208.49 KiB) Viewed 8131 times

On the bottom side of the mainboard, there's a clearly marked JTAG port. Since mainline OpenOCD doesn't have any PowerPC support, this was a whole side-quest. In the end, it turned out to be unnecessary, except for unbricking. NXP has a very nice document (AN4365) describing one of their JTAG-based debug protocols (either called OnCE or Nexus), but the SoC in this device unfortunately uses a completely different one (JTAG/COP), which is considered proprietary.

cop-proprietary.png (21.62 KiB) Viewed 8131 times

After borrowing a debugger from a friend, which implements JTAG/COP, I was finally able to read some registers, but the processor kept being reset every few seconds. That's when I put a jumper on the "DBG" header, which made the whole JTAG thing fairly irrelevant by spawning a bootloader debug menu/shell on the serial port:

Code: Select all

WATCHDOG DISABLED 
Found primary and mirror bbt

NO VALID FIRMWARE!
----------------------------------------------
LANCOM Systems Boot-Loader
Version 4.00.0001 / 12052014 / 132211
Copyright (C) 2012 LANCOM Systems GmbH
 Vendor: LANCOM Systems
 Oem:    
 Device: LANCOM L-321agn Wireless (R2)
 HwId:   NWAPP2
HW-Release: H, PLD-Release: --
CCB clock 133MHz, Core clock 400MHz, VCO clock 267MHz, DDR clock 133MHz LBC Clock 33MHz
Reset cause: 0

Debug-Adapter: absent
phystop: 08000000, ramfirm_end: 08000000
Minimum FW/LD version is 8.82/3.41
Active Firmware: 2
Bitrate = 115200.8N1
V<adr>/I/R/G/B/S/U/D/e/f/t/H/?>

Adr = 00000000, Len=0x80
00000000:  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 | ................
00000010:  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 | ................
00000020:  00 00 00 00 00 00 00 00  00 01 F9 E0 00 00 00 00 | ................
00000030:  00 10 00 00 00 10 73 30  00 00 8A 30 48 00 89 F4 | ......s0...0H...
00000040:  00 02 ED 10 00 00 00 04  00 00 00 00 00 00 00 01 | ................
00000050:  FE 00 00 00 FE 02 ED 10  FE 00 00 00 00 00 00 00 | ................
00000060:  34 2E 30 30 3F 00 00 00  00 FF 22 C1 00 00 00 02 | 4.00?.....".....
00000070:  00 02 ED 10 42 00 00 00  60 00 00 00 60 00 00 00 | ....B...`...`...

V<adr>/I/R/G/B/S/U/D/e/f/t/H/?>

Monitor help page
-----------------
 V<adr>  : Dump memory addresses <adr>
 G       : Start active firmware
 I       : Version Info
 U0,U1   : Upload file to Flash-ROM by XMODEM-/CRC
 R       : Reboot (Warmstart)

 Bx      : Modify outband bitrate

 E ...*  : Erase menu       * = action required
 F ...   : Firmware menu
 T ...   : Test menu

Thanks to this bootloader menu, I was able to dump the RAM contents (located at 0x0) over the serial port, which finally let me have a closer look at the (second stage) bootloader, so I could understand the Xmodem function. After some confusion it turned out to be the normal Xmodem protocol with support for CRC and 1k blocks, and you can simply upload a UPX file which is then written to flash (the bootloader keeps a history of two firmwares, you add a new one with the Xmodem upload and the older one is deleted). After some more reverse-engineering, I was running my own code!

aaa.png (9.92 KiB) Viewed 8131 times

[jn926]

... but only for a few seconds. Turns out, there's a watchdog connected to one of the GPIO pins (which also prevented me from using JTAG, earlier). With that in mind, I was able to port my usual 1000 lines of C code and have a rudimentary shell:

lolmon.png (8.47 KiB) Viewed 8131 times

Next stop: U-Boot

My goal was to boot Linux, but pushing 5-10 MiB through a 115200 baud serial port isn't fun, so I needed something that could use the Ethernet port, preferably without me having to write an Ethernet driver first. Luckily, U-Boot supports the Freescale MPC83xx chip family well enough that I was able to get it booting on there. Sending Ethernet frames worked almost immediately, but receiving had issues. All I got was a garbled mess, *roughly* in the shape of the frame I was sending:

mangled-frame.png (50.27 KiB) Viewed 8131 times

The issue turned out to be that the Ethernet PHY (AR8035) has internal delay logic, which one has to account for by specifying phy-mode = "rgmii-id" in the devicetree. After that, I had a working Ethernet stack. Time to boot Linux over TFTP!

On the Linux side, I started by untangling the old MPC8315E-RDB (reference design board) devicetree that had been committed back in 2008. Nowadays, it's strongly recommended to keep the SoC details in one file (mpc8315e.dtsi) and the board specific parts in another (mpc8315erdb.dts), so that different boards don't have to redefine stuff that will never change as long as the same chip is used.

On top of the new SoC devicetree, I had to add surprisingly little. The biggest part are the LED definitions: the board has six LEDs, four of which are bi-color, and footprints for adding even more.

But then came the hard part, figuring out how PCIe works. At first, it just didn't. The PCIe Link State Register (LTSSM) showed zero, i.e. not connected, even though the "lspci" utility in the bootloader menu showed whatever card I plugged in. The bug turned out to be in U-Boot's GPIO driver, which set all GPIO outputs to zero (low), as a side effect of a workaround for another chip generation. With that fixed, I could boot... to panic. Linux (or rather the SoC's bus hierarchy) wasn't happy with how I configured the pcie mappings in the devicetree. The OpenFirmware standard (which introduced devicetrees) has a "binding" that describes how to represent PCI(e) buses in a devicetree, but it wasn't very clear to me. Especially the "ranges" property gave me lots of ways to get it wrong.

gpu-setup.jpg (204.69 KiB) Viewed 8131 times

After a night of trying different address ranges which would create enough space to map the rather large BARs of a graphics card, and discovering the hardware register that control the translation of CPU bus (CSB) addresses to PCIe addresses, I tried something more normal, the Atheros WiFi card that was originally installed in this thing. To my amazement, it just worked; and a USB 3 card also (mostly) just works. I suspect the culprit is the long latency (or signal integrity issues) caused by the AliExpress mini-PCIe riser card that I use to connect full-sized PCIe cards.

[jn926]

A change of plans

I still wanted to connect a keyboard/mouse/screen and some mass storage, because that's just such a silly idea. Fortunately my USB card works, and I have an old DisplayLink dongle (with VGA output) and a USB hub.



The only remaining question is what to run on there. For ad-hoc testing, Buildroot is nice, but for desktop use it is rather lacking. Adélie Linux is one of the few remaining distros to ship ppc32 binaries, but it requires a PowerPC G3 or later. Debian's powerpcspe port requires the Signal Processing Extensions found in the e500 processor core, but the MPC8314E is based on the e300. Void might be an option, I haven't investigated it. Alpine can be bootstrapped for PowerPC, which I'll probably do.

Meet me at 38c3!

If you want one of these little computers, and you're at the Chaos Communication Congress at the end of this year, please let me know (either here or on fedi) before the 26th, so that I can take the right amount with me. It runs fairly small (< 1k lines) patchset on top of mainline U-Boot and Linux, and I'm in the process of upstreaming. Bring your own 12V power brick and RS-232 cable

EOF

[minki]

On the topic of Void for PowerPC, the core binaries like busybox, the package manager and compiler should be compatible with the minimal spec. Don't expect prebuilt software like SeaLion or Minetest to run though. Most programs in the repository were compiled targeting G3 or G4 machines.

See https://muellers-software.org/void-ppc-support/ for resources on running void on ppc.

[jn926]

See https://muellers-software.org/void-ppc-support/ for resources on running void on ppc.

Ah nice, thanks.

Right now I'm struggling with the USB stick that I got for this project, because it's extremely slow.

UPDATE: got a different USB stick, which is decently fast.

[jn926]

A lot happened at 38c3. I gave away all ten routers that I brought for that purpose. I made some progress bootstrapping Alpine Linux in 128 MiB and then hit the python wall: at 15000 lines (unicodeobject.c), the assembler (as) triggered the OOM killer; I "fixed" that by using -Os instead of -O2 for optimization, and also disabling PGO. But then there was deepfreeze.c, ten times as long, and I gave up on building python3. Meanwhile, nina (q66) bootstrapped Chimera Linux for ppc32, fixing a few toolchain bugs along the way. Three days in, there are like 2000 packages built.

It seems —for now— that PowerPC isn't dead.

[minki]

I made some progress bootstrapping Alpine Linux in 128 MiB and then hit the python wall: at 15000 lines (unicodeobject.c), the assembler (as) triggered the OOM killer; I "fixed" that by using -Os instead of -O2 for optimization, and also disabling PGO. But then there was deepfreeze.c, ten times as long, and I gave up on building python3.

The problem is the 128 MB here, I've run into this with my Nintendo 3DS before. What should be possible to avoid it is to run the process from swap, in other words, we can download more RAM. Since we have some rather good networking on this device, swap could be on an M.2 SSD or a RAM disk that is network attached at 1 Gbps using NFS. This would eliminate rootfs i/o usage for RAM swapping. Additionally, zram may be used to compress memory regions to make more effective use of the internal 128 MB.