====== A1508 ======
**Hobart 12m:**

Disk VSN: HAY-0055

Data volume at beginning: 1.4 GB (test recording + fringe check)

     * 00:11 UT : disc pos ~40 GB behind  -  Rxmon stop command returns "operation not permitted; no process killed" not sure what to do since everyone is in the WHS training... (Ellen)
     * 01:41 UT : Rxmon stopped and started. (Ellen)
     * 02:10 UT : swapped modules over. Looked like everything was running again fine but then the mark 5 went crazy and tried to record to both modules. We did a full power cycle 3 times to fix this. Scans between 050-0211 and 050-0239 are either bad or were missed. (Ellen)
     * Schedule resumed for scan 050-0241. (Ellen)
     * 09:46UT: mark 5 lost connection. Halted schedule tried SSReset to no avail, hard rebooted as superuser, seemed to fix the problem.  Schedule restarted 09:51, missed scans 050-0944b to 050-0956a   (Imogen)
     * 10:11UT: noticed that the time offset in the formatter (when issued mk5=dot? command) was more than 10ms.  In fact, it was 50ms.  However, this value fell with the next check and continued to fall until it was below 10ms. (Imogen)
     * 10:54UT: mark 5 keeps losing connection, not sure why (Imogen)
     * 11:49UT: scans 050-1146 to 050-1156 were lost due to attempt to heal the "ERROR m5 -104 mk5cn: time-out, connection closed" (restarted DIMino and fs, helped for now) (Liza)
     * 12:02-04UT: DIMino stopped, restarted it twice, affected scans 050-1202, 050-1204a (Liza)
     * 13:48-13:59UT: Problems with Mark5B, restarted DIMino repeatedly. Scans affected are 050-1348 to 050-1358. Meanwhile, clock jump occured, fmset synch performed 14:01;23 between scans 050-1400 and 050-1402 (Liza) Due to the Mark5 resetting, following scans were recorded to a wrong module UAO-0026/2000:
       * 1 a1508_hb_050-1356+                                  
       * 2 a1508_hb_050-1358                            
       * 3 a1508_hb_050-1400                            
       * 4 a1508_hb_050-1402 
     * 16:45 restarted DIMino, fs, schedule. Scans 050-1645 and 050-1647 were affected (Liza)
     * By the end, it recorded 300GB less due to repeated ERROR m5 -104 mk5cn: time-out, connection closed and attempts to recover it (Liza)

**Katherine 12m:**

Disk VSN: HOB+0124

Data volume at beginning: 1.12 (test recording + fringe check)

     * 22:35 UT : clkoff returned a value ot ~E-001, while maserdelay was fine. "counter" didn't help, neither did "fmset" (Ellen)
     * Scans 049-2235 to 049-2245 are bad.
     * 22:47 UT : restarted the DBBC session and reconfigured, clkoff is still giving value of ~E-002. I open fmset again and the values all matched up, but I synced it any way in the hope of fixing the clkoff issue. But this actually un-synced the 3 numbers... "counter" didn't fix it this time either. (Ellen)
     * 23:00 UT : Jamie walks in the room and suddenly the it decides to fix itself >.< (Ellen)
     * 23:02 UT : Schedule wouldn't start properly, then the field system wouldn't respond to any commands. Tried to kill it but it wouldn't die properly so I couldn't restart it. Dave came to the rescue and killed it properly so I could restart the schedule at 23:21 UT. For most of this time the mark 5 was recording non-stop. Missed scans 049-2247a to 049-2326. (Ellen)
     * Due to continuous recording, disc pos now ~53 GB ahead of schedule. (Ellen)
     * Scan 049-2327 completed successfully. (Ellen)


**Yarragadee 12m:**

Disk VSN: HOB+0121

Data volume at beginning: 8369 GB

  * We had some initial trouble with the module or mk5 when it refused to do a test record. The pointers were noticed to be at ~4TB rather than the expected ~8TB. The module eventually came good after:
      * moving the module to the other bank
      * rebooting mk5yg
      * running SSReset
      * restarting DIMino several times,
      * running "mk5=recover=2"
  * The DBBC issue causing problems in S band is ongoing

  * The wind monitor has not been able to connect from 22:00 UT to 00:30 UT (and counting). (Ellen)
  * By the end, it recorded 100GB less (Liza)

{{tag>}}


~~DISCUSSION~~