I started with the 8.04.1 Xubuntu Alternate CD, available here.
This doesn't seem to have an option to install a command line system, which surprised me.
So I installed Xubuntu 8.04.1. Which does actually run on 128MB RAM (with default "auto" swap partition size).
However, there's never any need to boot into Xubuntu at all…
NB: If you want an Ubuntu 8.10-based system (Intrepid): look here