11 Oct 2004 M0 (CIMT) MMC post-mortem M0 failed (date ~09/27/04) . The compact flash could not be detected. gosiam worked; but would show corrupt directories... 3 (?) rest attempts after resets, comms wouldn't come back; gosiam did not work Notes: Can still had pressure. on boot,CF detected and app ran. able to mount CF and see siam dir exitNode halt cardmgr[58]: + ./ide: fuser: command not found power cycle made it thru init and went to sleep cpuAwake 3600 (10:22) exitNode fsck.ext2 w/o repair (may have repaired first one) fsck.ext with repair open can CF held in place with plastic sleeve over bolt (couldn't seat further) took photos deep grooves on housing O-Ring seal surfaces (not likely related) Talked to Mark, Brent: - Pager reset doesn't remove power from CPU; may not have cycled power to CF - (what if Linux rebooted-> module reloaded? (does module cycle power on startup?) - there is a utility called hdparm [utility to send commands to ide driver; power down/up, etc)] - failure is consistent w/ CF HW failure or with CF going to sleep (this would be fixed by power cycling CF via hdparm), or with ide driver/CF firmware interaction (possible, since ide driver is cerf specific) - if it happens again, we might try shutting down power only via hdparm, unloading/reloading ide module. cpuAwake 3600 ... built hdparm on Bufflhead and zmodem'd it to MMC used it to read disk info uploaded /mnt/hda/logs/messages.bak uploaded /mnt/hda/logs/syslog.bak noted some nicks in wire harness where it contacts the chassis wiring vias uploaded /mnt/hda/siam/logs/node.200406012204 Tues, Oct 12 cpuAwake 36000 (10 hrs) Began uploading all logs via zmodem [too slow] exitNode ifconfig connect eth cable ifconfig eth0 up looked at hosts went to /etc/ su console froze after entering root passwd MMC lights (RGY,heartbeat) still lit ctlz,c,x,q,s start/stop hperterm connection restart hyperterm power down power up exitNode restart capture kill pppd longhaul modified modules,modules.conf for eth halt power cycle (plugged in eth cable during boot) exitNode kill pppd longhaul (as root) ifup eth0 failed try various cables... kill manage log processes, boa ... ifdown -a ifup -a set SIAM_AUTOSTART=FALSE halt power cycle ifswitch-to-dhcp comment out ifdown eth0 lines in /etc/init.d/rc.local halt reboot sequence at start: if ide.opts (sourced by ide) which sets $PARTS and returns with $DO_FSCK='y' (and others) cardmgr->/etc/pcmcia/ide->(sources)/etc/pcmcia/shared->ide(calls)start()->(shared)(calls)add_part()-> (shared)(calls)add_blkdev()->calls fsck -Ta /dev/hda and mount may need to modify /etc/pcmcia/ide.opts for CF to run fsck at boot related files: /etc/pcmcia/ide /etc/pcmcia/shared /etc/pcmcia/ide.opts ran fsck.ext2 -va (all OK)(-a maps to -p: repair no questions)(outputs hda:hda1) - fsck -Ta doesn't work b/c -T is not supported and fsck doesn't recognize type auto - maybe can add "-t ext2" (no that won't work, maybe setting FSTYPE will though)to options in ide.opts to fix fsck auto problem ran mount /dev/hda1 /mnt/hda outputs( hda: hda1 hda: hda1 hda: hda1 ) modified /etc/pcmcia/ide.opts: case "$ADDRESS" in *,*,*,1) echo "ide.opts configuring $ADDRESS partition 1" INFO="CF IDE setup" DO_FSTAB="n" ; DO_FSCK="y" ; DO_MOUNT="y" FSTYPE="ext2" OPTS="" MOUNTPT="/mnt/hda" ;; *,*,*) echo "ide.opts configuring $ADDRESS" PARTS="1" # Card eject policy options NO_CHECK=n NO_FUSER=n ;; esac works OK, but still errors out b/c fsck.auto error modified fstab (changed fstype from auto to ext2 for /dev/hda1): # /dev/mtdblock3 / jffs2 defaults 0 0 tmpfs /mnt/ramfs tmpfs defaults 0 0 proc /proc proc defaults 0 0 devpts /dev/pts devpts mode=0622 0 0 /dev/hda1 /mnt/hda ext2 defaults 0 0 #tempest:/vol/vol0/users/oreilly /home/oreilly/tempest nfs bg,soft,nosuid,nodev,exec OK: fsck is executed before mounting looking thru kernel source to find out if/when/how resets take place: - relevant files: ide.c, ide-disk.c, hdparm.c (not part of intrinsyc src), ide-cs.c - the driver associated w/ /dev/hda1 is ide-disk.c. The guts of it is in ide.c - hdparm uses ioctl to control devices; it passes a file descriptor (e.g. /dev/hda1) to the driver (ide-disk.c), which does the operation. - the ide interface is reset by callling ide_do_reset w/ args (drive,0) - ide.c::do_reset1 is the entry point to reset code. it sez: /* * do_reset1() attempts to recover a confused drive by resetting it. * Unfortunately, resetting a disk drive actually resets all devices on * the same interface, so it can really be thought of as resetting the * interface rather than resetting the drive. * * ATAPI devices have their own reset mechanism which allows them to be * individually reset without clobbering other devices on the same interface. * * Unfortunately, the IDE interface does not generate an interrupt to let * us know when the reset operation has finished, so we must poll for this. * Equally poor, though, is the fact that this may a very long time to complete, * (up to 30 seconds worstcase). So, instead of busy-waiting here for it, * we set a timer to poll at 50ms intervals. */ Wayne sez: - It is up to SW to reset the CF at boot time; the HW to do so was removed from sidearm 1.3 b/c sleep manager (?) is supposed to do it (yank some GPIO mapped to CF_RESET) after sleep (?) and something should do it at boot as well. Does CF reset get yanked? - cs.c (card services) registers with power management. - when a power management event (PM_SUSPEND,PM_RESUME, etc.) occurs, pm sends out a message to the registered callback. - when cs receives the PM_SUSPEND it calls pcmcia_suspend_socket (socket_table [i]); - pcmcia_suspend_socket calls send_event(s, CS_EVENT_PM_SUSPEND, CS_EVENT_PRI_LOW), which notifies all CS managed devices that a PM_SUSPEND has occurred, and then calls suspend_socket(s). - suspend_socket sets s->socket=socket and returns s->ss_entry->suspend(s->sock) - s is a socket_info_t* (cs_internal.h), it's ss_entry is a pccard_operations* (defined in ss.h) which defines pccard operations: struct pccard_operations { int (*init)(unsigned int sock); int (*suspend)(unsigned int sock); int (*register_callback)(unsigned int sock, void (*handler)(void *, unsigned int), void * info); int (*inquire_socket)(unsigned int sock, socket_cap_t *cap); int (*get_status)(unsigned int sock, u_int *value); int (*get_socket)(unsigned int sock, socket_state_t *state); int (*set_socket)(unsigned int sock, socket_state_t *state); int (*get_io_map)(unsigned int sock, struct pccard_io_map *io); int (*set_io_map)(unsigned int sock, struct pccard_io_map *io); int (*get_mem_map)(unsigned int sock, struct pccard_mem_map *mem); int (*set_mem_map)(unsigned int sock, struct pccard_mem_map *mem); void (*proc_setup)(unsigned int sock, struct proc_dir_entry *base); }; Tim says that the GPIO to reset the CF (GPIO_CF_RESET; sa1100_cerf.c) is asserted at boot; It is not know if/when/how the signal is propagated thru the CPLD. There is an extender card somewhere that we might use to check (and ask Wayne). An aside: Tim mentioned that some piece of code in sa1100_cerf.c used only during plugin/eject was ifdef'd out. Thursday 10/14 Wayne: The GPIO_CF_RESET signal is asserted by the Linux loader (Tim wrote/modified parts of this code), which runs in between the bootloader and Linux proper, setting up basic hardware (which Linux assumes to have been done). This signal is propagated through the FPGA, but it is not known how long it is asserted (but the code asserts it and deasserts it one instruction later). It could be that it is not asserted long enough. However, it is not certain that this would fix the CF problem, since it does not power cycle the CF. -> check mfg data sheet to see how long reset needs to be asserted and measure actual time. There is also a power switch, CF_POWER_ON, that is connected to a GPIO on the SA1110. Currently, however, although that signal may be wiggled by the SA1110, the FPGA just asserts it at start up and ignores changes that occur at its input from the SA1110. -> verify the behavior of CF_POWER_ON on both sides of the FPGA at boot and when the GPIO is wiggled. -> see if hdparm can do anything We may not be able to power cycle The original CF tests were just large read/writes from/to the CF, w/o sleeping. -> Do Tim's CF test w/ sleep to try to reproduce the condition The kernel we are running is old (2.4.10; latest version 2.4.20+), and a number of filesystem bugs have been fixed in between. It would be good to upgrade the linux version we use; J9 has strong dependencies on glibc, and doesn't work with 2.4.20; Tim is working on making tweaks (downrev glibc) to make it go. -> check the bug reports on filesystem, ide, etc. to see what linux kernel bugs may exist that could affect this condition. -> check mfg data sheet to see how long reset needs to be asserted and measure actual time. Need: data sheet, sidearm, extension card?, logic analyzer -> verify the behavior of CF_POWER_ON on both sides of the FPGA at boot and when the GPIO is wiggled. Need: sidearm, logic analyzer, extension card?, GPIO wiggler -> see if hdparm can do anything -> Do Tim's CF test w/ sleep to try to reproduce the condition -> check the bug reports on filesystem, ide, etc. to see what linux kernel bugs may exist that could affect this condition. [ Tues, Oct 19 ] - set up Sidearm, logic analyzer, CF extender - CF_RESET is asserted for 150 ms once or twice during boot (includes soft reset via SW1 ~ pager reset) - CF_POWER_ON is asserted when power is applied to the sidearm, and does not change during reset