Hobart 12m:

Disk VSN: HAY-0055

Data volume at beginning: 1.4 GB (test recording + fringe check)

• 00:11 UT : disc pos ~40 GB behind - Rxmon stop command returns “operation not permitted; no process killed” not sure what to do since everyone is in the WHS training… (Ellen)
• 01:41 UT : Rxmon stopped and started. (Ellen)
• 02:10 UT : swapped modules over. Looked like everything was running again fine but then the mark 5 went crazy and tried to record to both modules. We did a full power cycle 3 times to fix this. Scans between 050-0211 and 050-0239 are either bad or were missed. (Ellen)
• Schedule resumed for scan 050-0241. (Ellen)
• 09:46UT: mark 5 lost connection. Halted schedule tried SSReset to no avail, hard rebooted as superuser, seemed to fix the problem. Schedule restarted 09:51, missed scans 050-0944b to 050-0956a (Imogen)
• 10:11UT: noticed that the time offset in the formatter (when issued mk5=dot? command) was more than 10ms. In fact, it was 50ms. However, this value fell with the next check and continued to fall until it was below 10ms. (Imogen)
• 10:54UT: mark 5 keeps losing connection, not sure why (Imogen)
• 11:49UT: scans 050-1146 to 050-1156 were lost due to attempt to heal the “ERROR m5 -104 mk5cn: time-out, connection closed” (restarted DIMino and fs, helped for now) (Liza)
• 12:02-04UT: DIMino stopped, restarted it twice, affected scans 050-1202, 050-1204a (Liza)
• 13:48-13:59UT: Problems with Mark5B, restarted DIMino repeatedly. Scans affected are 050-1348 to 050-1358. Meanwhile, clock jump occured, fmset synch performed 14:01;23 between scans 050-1400 and 050-1402 (Liza) Due to the Mark5 resetting, following scans were recorded to a wrong module UAO-0026/2000:
  • 1 a1508_hb_050-1356+
  • 2 a1508_hb_050-1358
  • 3 a1508_hb_050-1400
  • 4 a1508_hb_050-1402
• 16:45 restarted DIMino, fs, schedule. Scans 050-1645 and 050-1647 were affected (Liza)
• By the end, it recorded 300GB less due to repeated ERROR m5 -104 mk5cn: time-out, connection closed and attempts to recover it (Liza)

Katherine 12m:

Disk VSN: HOB+0124

Data volume at beginning: 1.12 (test recording + fringe check)

• 22:35 UT : clkoff returned a value ot ~E-001, while maserdelay was fine. “counter” didn't help, neither did “fmset” (Ellen)
• Scans 049-2235 to 049-2245 are bad.
• 22:47 UT : restarted the DBBC session and reconfigured, clkoff is still giving value of ~E-002. I open fmset again and the values all matched up, but I synced it any way in the hope of fixing the clkoff issue. But this actually un-synced the 3 numbers… “counter” didn't fix it this time either. (Ellen)
• 23:00 UT : Jamie walks in the room and suddenly the it decides to fix itself >.< (Ellen)
• 23:02 UT : Schedule wouldn't start properly, then the field system wouldn't respond to any commands. Tried to kill it but it wouldn't die properly so I couldn't restart it. Dave came to the rescue and killed it properly so I could restart the schedule at 23:21 UT. For most of this time the mark 5 was recording non-stop. Missed scans 049-2247a to 049-2326. (Ellen)
• Due to continuous recording, disc pos now ~53 GB ahead of schedule. (Ellen)
• Scan 049-2327 completed successfully. (Ellen)

Yarragadee 12m:

Disk VSN: HOB+0121

Data volume at beginning: 8369 GB

• We had some initial trouble with the module or mk5 when it refused to do a test record. The pointers were noticed to be at ~4TB rather than the expected ~8TB. The module eventually came good after:
  • moving the module to the other bank
  • rebooting mk5yg
  • running SSReset
  • restarting DIMino several times,
  • running “mk5=recover=2”
• The DBBC issue causing problems in S band is ongoing

• The wind monitor has not been able to connect from 22:00 UT to 00:30 UT (and counting). (Ellen)
• By the end, it recorded 100GB less (Liza)