Verement's Journal

30-Aug-1999

I received notice that my empeg is ready to be ordered! Using my registration number, I placed an order for a 14G blue display unit. Within a few hours, however, I was notified that the 14G drives were in scarce supply, so I downgraded my order to 10G rather than get a two-drive 10G+4G.

01-Sep-1999

I received confirmation that my order has been dispatched, to be shipped the first week of September.

02-Sep-1999

I'm looking forward to doing some development on the empeg, so I'm looking into building a cross-compile environment for my i586 system.

The suggested starting point is a page on Intel's web site describing the process.

I can't seem to find a number of files referenced by Intel's documentation, so I'm punting a little and will try to use gcc 2.95.1 instead of egcs, as there is an ARM patch for gcc 2.95 and binutils 2.9.1.0.25 should be usable as-is.

03-Sep-1999

I received notice that my empeg has been shipped!

I successfully built binutils and gcc, but I'm having problems building glibc 2.1.1. I'm getting some undefined symbols during linking.

05-Sep-1999

I gave up on gcc 2.95.1 and decided to use egcs 1.1.2 instead, again with an ARM patch. This time glibc built without any problems. I'm still using binutils 2.9.1.0.25 unmodified.

07-Sep-1999

It has arrived! My unit has serial number 00100.

After listening to the free music sample several times, I broke down and upgraded the 95 partition of my 95/NT box to 98 so I could install the beta6 emplode software. I plugged in the empeg using the USB cable and played around with it a bit. I uploaded a few 80s songs to the player and verified everything worked.

At this point I also wanted to install the beta6 developer image, but had some problems with my serial port. The upgrade will not use USB, and I already had a device connected to COM1. Win 98 didn't know about my second serial port, but even after adding it with Add Hardware, I still could not get emplode to see my player connected to it. So, I unplugged my other device and used COM1 after all. The upgrade went smoothly after this.

I had some trouble figuring out how to get a response from the serial connection once the player was running. First was finding the right serial parameters: 115200 8N1, no handshaking. I forgot about the handshaking part and found the player unresponsive to anything I typed because my terminal assumed RTS/CTS handshaking. (Q: why no handshaking?)

Typing q (return) brings a root shell.

The environment is extremely minimal:

bash-2.01# ls /etc /sbin /bin /usr/bin /lib /usr/lib
/etc:
fstab  ld.so.conf  terminfo

/sbin:
fdisk  fsck  fsck.ext2  halt  init  insmod  mke2fs  reboot  shutdown  update

/bin:
awk    cmp  diff    ln     mkswap  nice  rmdir  rz    swapoff  tar
bash   cp   echo    ls     more    od    ro     sed   swapon   umount
cat    dd   gunzip  mkdir  mount   ps    rw     sh    sync     vi
chmod  df   gzip    mknod  mv      rm    rwm    stty  sz

/usr/bin:
grep  ldd

/lib:
ld-2.1.so          libdl-2.1.so      libm.so.6          libreadline.so.2.1
ld-linux.so.2      libdl.so.2        libncurses.so.4    libutil-2.1.so
libSegFault.so     libe2p.so.2       libncurses.so.4.2  libutil.so.1
libc-2.1.so        libe2p.so.2.3     libproc.so.1.9.0   libuuid.so.1
libc.so.6          libext2fs.so.2    libpthread-0.8.so  libuuid.so.1.2
libcom_err.so.2    libext2fs.so.2.4  libpthread.so.0
libcom_err.so.2.0  libm-2.1.so       libreadline.so.2

/usr/lib:
ld.so.1

There are some interesting things in /proc:

bash-2.01# ls /proc
1    audio      empeg_id     empeg_usb    kcore    mounts      stat
161  bus        empeg_ir     filesystems  kmsg     net         swaps
2    inputline  empeg_power  fs           loadavg  partitions  tty
3    cpuinfo    empeg_rds    ide          locks    scsi        uptime
4    devices    empeg_state  interrupts   meminfo  self        version
42   dma        empeg_therm  ioports      misc     slabinfo

The disks are mounted read-only by default, however two scripts rw and rwm will make all the partitions or just the music partition(s?) writeable, respectively. I forgot a few times to call ro later before resetting the unit, causing me to need to fsck the partitions upon reboot.

I had some trouble getting the mount and df commands to work properly, so I did:

# ln -s /proc/mounts /etc/mtab

This enabled df to work:

bash-2.01# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/root                31727     23115      6974  77% /
/dev/hda4              9384146     73713   9310433   1% /drive0

(/dev/root is /dev/hda1.)

Here is the full partition map according to fdisk:

Disk /dev/hda: 16 heads, 63 sectors, 19485 cylinders
Units = cylinders of 1008 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1             1        66     33232+  83  Linux
/dev/hda2            67       132     33264   83  Linux
/dev/hda3           133       165     16632   83  Linux
/dev/hda4           166     19485   9737280   83  Linux

According to the folks at empeg, /dev/hda2 is unformatted scratch space which is protected from being clobbered by any upgrades.

However, after formatting with mke2fs and using the partition a bit, a later fsck on the formatted but unmounted partition yielded:

bash-2.01# fsck /dev/hda2
fsck: memory violation at pc=0x020013b4, lr=0x4005f7d8 (bad address=0x00000004, code 3)

pc : [<020013b4>]    lr : [<4005f7d8>]
sp : bffffd74  ip : bffff6ac  fp : bffffde8
r10: 400fe134  r9 : 0200a6e0  r8 : 0200a3d4
r7 : 0200a768  r6 : 00000000  r5 : 00000000  r4 : bffffd7c
r3 : 00000000  r2 : bffffd86  r1 : 02002203  r0 : 00000009
Flags: Nzcv  IRQs on  FIQs on  Mode USER_32  Segment user
Control: C825917D  Table: C825917D  DAC: 00000015
Function entered at [<02001328>] from [<02001dec>]
  r7 = 0200a3f0
  r6 = 00000000
  r5 = 0200a6dc
  r4 = 00000000
Function entered at [<02001cc0>] from [<4003995c>]
  r9 = 02001eb8
  r8 = 400fd0a0
  r7 = 02001cc0
  r6 = 400217c0
  r5 = 00000002
  r4 = bffffe54
Function entered at [<4003982c>] from [<020008c8>]
Segmentation fault

This seems bad. I do not get this output on /dev/hda1 or /dev/hda4, however I do sometimes get:

do_page_fault(): HACK TRIGGERED

I don't know what /dev/hda3 is used for... (swap?)

I found a Debian ARM root filesystem tarball with a minimal Debian setup from ftp://ftp.jimpick.com/pub/debian/arm/image/. I unpacked this into /drive0/local/root (it was too big for /dev/hda2) and set up a symlink from /usr/local to /drive0/local. I also added a script /usr/local/sbin/debian which does:

#! /bin/sh

cd /usr/local/root
exec usr/sbin/chroot . bin/login

I've done some various tweaking to the Debian root, including fixing permissions on /dev/null, /dev/zero, /tmp, and /var/tmp. I also added myself a non-root user account to play with.

The unit's date was not set; I set it. I also set a hostname.

When I run the Debian version of fsck on /dev/hda2 as above, it doesn't crash like above; I get normal output.

I am looking for a way to add PPP support, but it looks like this may require help from the kernel...

Btw, I'm having intermittent problems uploading files to the device using rz; for some reason the reciprocal sz on my end is randomly dumping core in the middle of transfers. My solution is to split -b 200000 all files larger than this size, transfer them individually (in a batch), and cat them back together on the empeg. This is cheap insurance that I can resume an aborted transfer without having to start completely over. (I thought zmodem supported automatic transfer resume?)

08-Sep-1999

The unit's date has drifted by several hours, but I'm not sure if this was caused by resetting the unit (removing power).

I installed the Debian gcc package (2.91.60-5.1) and prerequisites. I also installed the libc6-dev package (2.1.2-0pre7).

I ran out of memory a few times so I added 64M of swap space (probably overkill) as /var/tmp/linux.swap to give me enough room to compile packages.

Still no luck with PPP. I don't know how to add kernel support.

I tried a display test program using mmap'd /dev/display, but I don't remember how to blit the changes.

09-Sep-1999

Hugo has given us a PPP kernel upgrade, answered my questions about blitting the display image and controlling the front status LED, and offered www.empeg.org as a potential place to start putting up developer info...

The PPP kernel is: http://www2.empeg.com/upgrades/tools/pppkernel.upgrade

Using the PPP kernel, I'm getting lots of messages similar to:

kmod: failed to exec /sbin/modprobe -s -k net-pf-1, errno = 2

I copied /usr/local/root/sbin/modprobe to /sbin to silence these messages.

I successfully ifconfig'd the lo interface and can ping it. (I had to chmod u+s /bin/ping.)

There is no /etc/hosts file, so I'm making one. I also fixed /etc/resolv.conf.

I made /dev/console a link to ttyS1, just as it is in the real root.

Now I need to install the kernel headers so I can build PPP.

First I want to build zsh because I'm getting tired of bash.

Now I need patch.

I see:

do_page_fault(): HACK TRIGGERED

in the middle of the zsh configure run.

Now I need libncurses4-dev, yodl, perl5, dpkg-dev, tetex-bin and tetex-base, and dpkg-perl.

Apparently the empeg's default /sbin/init is not reaping processes that it ends up inheriting. This is leaving lots of <defunct> processes lying around.

Now I need xlib6g and ed.

I have unresolved dependencies in tetex-bin (xlib6g) but I have enough now to finish the zsh build. (I just needed makeinfo and texi2html.)

I have finished building zsh! Now, to clean up my environment...

Now I need ncurses-bin.

Now I am ready to build the Debian ppp package. A first-run pass is giving me errors about missing <asm/proc/param.h> headers. I have the kernel-headers package installed, but /usr/include/asm/{arch,proc} are just empty directories instead of symlinks. I'm not sure which directories are appropriate but I'll try arch-ebsa110 and proc-armv.

The stock ppp debian/rules is complaining about missing <security/pam_appl.h>. I don't know where this is, and I'm not certain I care about PAM, so I will need to undefine USE_PAM.

I gave up on this idea and went ahead and installed libpam0g-runtime, libpam0g, and libpam0g-dev.

The ppp package(s) are built! There are actually two versions, ppp and ppp-pam. I'm going to install the former.

Now to configure PPP...

Before I do that, I need at least one network service or there won't be much to do except ping the unit. How about ssh... There is no pre-built Debian package, so I'm on my own again. I'm building it now.

Before I finished that, I played with the screen some more. I succeeded in writing to the screen, and wrote some utility programs to capture/blit the display and also turn it off and on.

I got tired of exiting the chroot environment every time I need to rz, and since rz doesn't work for some reason in the Debian chrooted environment, I built and installed the Debian lrzsz package.

For kicks, I ran Jay Carlson's dhrystone benchmarks. I get 280k dhrystones/sec using egcs 2.91.60. (My P100 gets 142k d/s using gcc 2.7.2.3.)

Back to ssh... Now I need libsocks4. I wonder if lack of xlibs will be a problem. Now I need gmp2.

10-Sep-1999

I have a working ssh package, and I managed to bring PPP up. Now I can have simultaneous interactive sessions and usable file transfers...

More Debian package maintenance. I rebuilt the util-linux package because it was not picking up fdisk or cfdisk under ARM. I can't seem to install man-db because there is a conflict with libc6.

I need to replace /sbin/init ASAP so these defunct processes can get reaped.

Something is wrong with syslogd, and this could probably explain some problems with ssh as well: they can't create UNIX domain sockets. I suspect the kernel has not been built with UNIX domain socket support...

12-Sep-1999

I have been distracted by a new computer. I switched the PPP link over to this computer to free up the serial port on my server, which would rather be plugged into the UPS anyway.

I managed to replace /sbin/init with a version that (optionally) launches the Debian environment, and can even run straight into PPP via runlevel 3. It is designed to timeout if no input occurs so that the unit can still run as a standalone player in its intended design.

Here is the new /sbin/init. I renamed the original file to /sbin/init.empeg. Note that I had to copy /bin/sleep from the chrooted environment to the real root.

#! /bin/sh

PATH="/sbin:/usr/sbin:/bin:/usr/bin"
export PATH

exec 0</dev/ttyS1 1>/dev/ttyS1 2>&1
echo
echo "custom init starting"

trap 'echo "timeout: running stock init"; exec /sbin/init.empeg' USR1
parent=$$
( sleep 5; kill -USR1 $parent ) &

echo -n "boot: "
read boot
trap - USR1

case $boot in
    ppp)
        echo "booting ppp"
        runlevel=3
        ;;
    *)
        echo "booting console"
        runlevel=2
        ;;
esac

echo "mounting /proc"
mount /proc

echo "checking filesystems"
fsck -A

echo "mounting filesystems"
mount -n -t ext2 -o remount,ro /dev/hda1 /
mount -n -t ext2 -o rw /dev/hda4 /drive0

echo "running Debian init from chroot"
cd /usr/local/root
exec usr/sbin/chroot . sbin/init $runlevel

I would like to run the player in the background but I need to find a way to keep it from spamming the serial port.

According to Hugo, /dev/hda3 is used for persistent state so it is not a good candidate for scratch space. I will need to find another way to manage creating a small root fs, probably by using /dev/hda2 although that makes modifying /dev/hda1 from the chroot environment difficult (I've been using /dev/hda2 as a staging area for these mods.)

I was running xntpd to keep the clock current, but I found it was eating too much memory, so I've disabled it.

15-Sep-1999

I recreated the following devices under the chrooted environment:

crw-rw-rw-   1 root     root     245,   4 Sep 15 17:58 /dev/audio
crw-rw-rw-   1 root     root     244,   0 Sep  8 15:07 /dev/display
crw-rw-rw-   1 root     root     245,   3 Sep 15 12:42 /dev/dsp
crw-rw-rw-   1 root     root     246,   0 Sep 15 17:59 /dev/empeg_state
crw-rw-rw-   1 root     root     242,   0 Sep 15 17:55 /dev/ir
crw-rw-rw-   1 root     root     245,   0 Sep 15 17:58 /dev/mixer
lrwxrwxrwx   1 root     root            6 Sep 15 18:05 /dev/radio -> radio0
crw-rw-rw-   1 root     root      81,  64 Sep 15 18:05 /dev/radio0
crw-------   1 root     root     248,   0 Sep 15 17:59 /dev/rds0
lrwxrwxrwx   1 root     root            4 Sep 15 18:05 /dev/usb -> usb0
crw-rw-rw-   1 root     root     243,   0 Sep 15 17:59 /dev/usb0

I discovered that Jim Pick has some Debian X and ssh packages. I haven't tried his ssh package, but xlib6g will be useful in resolving some dependencies, so I installed it along with xlib6g-dev and xfree86-common.

16-Sep-1999

I wrote a program to stream-capture the display at specified intervals, and used it to create the animated visualization graphic on the home page of this site. The steps were basically:

  1. capture the display at 1/10 second intervals, streaming out to a file
  2. run the player while manually cycling through the visuals to capture something interesting :-)
  3. split the resulting file into 2048-byte screen image chunks
  4. convert each screen image into a more accessible PNG format (I wrote a program to do this)
  5. use ImageMagick convert to merge all the PNG files into an animated GIF
  6. use the GIMP to open and re-save the animated GIF file (to get much better compression)

I've been looking into what it might take to run PPP over USB instead of the slower serial port, but I haven't quite got all the details worked out yet. I will definitely need to upgrade my Linux desktop kernel to get any USB drivers. The rest probably involves learning the ins and outs of USB.

With all the proper device files in place, I managed to run the ARM version of the Xaudio player. I had to ln -s /lib/ld-linux.so.2 /usr/lib/ld.so.1 in the chrooted environment for this executable to run, however. The process uses about 27% of the CPU on average, and does have a tendency to skip/stutter, even if I turn swapping off and change the process to use real-time scheduling.

21-Sep-1999

I have connected my empeg to a wireless CDPD modem, and am running PPP at 19.2Kbps. However, I appear to be having some problems due to the lack of any hardware flow control support from the empeg. The symptom is that the modem will sometimes reset during excessive transfers, creating a noticeable delay and inconvenience.

25-Sep-1999

I have constructed a replacement root filesystem in /dev/hda2 based on the chrooted environment I've been working in, but also including the components from the real root, most particularly the /empeg directory. The idea I have is to replace the root filesystem on /dev/hda1 with this new root.

After some pitfalls and several reinstallations of the beta6a developer's upgrade file to recover from catastrophe, I managed to replace the root filesystem. Important safety tips: When creating the staging filesystem on /dev/hda2, specify the number of blocks from the /dev/hda1 partition; the two partitions are not exactly the same size. Also, when overwriting /dev/hda1, take care not to be running any programs from that filesystem, and reboot ASAP. I did not try working from a RAM disk, but I may experiment with it.

While replacing the root filesystem was successful, I am now in a state where it is hard to run the player. I have found it impossible to run the player and PPP at the same time, as they both want to use the serial port. Even Hugo's suggested /empeg/bin/player -s- to run the player without starting the serial protocol does not seem to help. Perhaps this only works in the consumer version?

16-Oct-1999

I had my empeg installed today!

I took my Ford Taurus to a professional installer. This was their second empeg installation, so I expected they would get it mostly right. They did do a good job, but to my dismay they ignored the extra ground leads which came with the empeg and which were supposed to be connected at the amplifier.

I'm still using my factory speakers, but the new amplifier makes the empeg sound rather exceptional. I'm using the Kenwood KAC-848 on the advice of some people who reported noise problems with some other amps. I did not encounter any problems until it rained: then I heard significant scratching, crackling and popping. However, I caused these noises to stop by installing the special ground leads to the amplifier myself as they should have been.

Despite others' reports to the contrary, my radio reception is quite good. I don't even have installed the antenna amplifier that now comes with the empeg. The RDS feature also works beautifully for the few stations in my area that use it.

I do have one complaint: when the empeg is left in radio mode after turning off the engine, when I turn the engine on again there is a loud pop that is almost deafening, if not at least quite frightening, as the empeg boots up. Normally the only time I hear a pop is when turning off the engine, regardless of the empeg's mode, due to the amplifier itself. This seems to be something different.

18-Jan-2000

In the midst of some serious Debian hacking, I managed to hose some important shared libraries, making my empeg unbootable. Unfortunately I don't have a recent backup of the partition, so I am a bit peeved.

I've installed beta8c for the time being. It has rearranged my partitions thusly:

Disk /dev/hda: 16 heads, 63 sectors, 19485 cylinders
Units = cylinders of 1008 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1             1        66     33232+   5  Extended
/dev/hda2            67       132     33264   83  Linux
/dev/hda3           133       165     16632   10  OPUS
/dev/hda4           166     19485   9737280   83  Linux
/dev/hda5             1        49     24633   83  Linux
/dev/hda6            50        66      8536+  82  Linux swap

Making sense of this, the swapfile which used to live in the root directory of /dev/hda1 is now a symlink to /dev/hda6 -- a clever move. It seems to me that /etc/fstab should now list /dev/hda5 as the root filesystem, but alas it still says /dev/hda1.

Now I need to figure out what to do with my poor Debian setup.

25-Mar-2000

Toby from Prolux has informed me that they will be opening a new web site soon containing detailed developer bits with respect to the empeg visuals. In connection with that effort, I have begun to use an improved technique for capturing the existing empeg visuals to make high-quality GIF animations.

Unfortunately I'm not yet able to capture all of the visuals this way.

Contributed by Rob Leslie