Hobart 12m:

Disk VSN: HOB+1011

Data volume at beginning: 9193 GB

• Experiment started OK (Ross).
• 8:27 UT : disc position ~50 GB behind - Rxmon restarted. (Ellen)

Katherine 12m:

Disk VSN: HOB+0124

Data volume at beginning: 12406 GB

• Experiment started OK (Ross).

Yarragadee 12m:

Disk VSN: HOB+0130 (changed to HOB+0102 part way through, details below)

Data volume at beginning: 0 GB (missing 6 GB)

• Missed first scan due to module swap (Ross).
• 19:43:11 UT: Wind stowed, missed scans 053-1943 through 053-1956b (Ross).
• 20:18:55 UT: Wind stowed, missed scans 053-2018 through 053-2036 (Ross).
• 21:11:08 UT: Wind stowed, missed scans 053-2111 through 053-2119 (Ross).
• 07:39 UT : noticed that the autocorrelation was terrible. So was the next scan. Previous scans might also be bad, I just didn't notice. When I tried to investigate, the field system reported that the connection was lost to the mark 5. Bad scans (at least) from 054-0738 to 054-0742. (Ellen)
• 07:44 UT schedule halted, restarted DBBC and the field system. This didn't help. Called Imogen, she said the pointers were wrong and so we need to change the module. Test recording went ok so schedule was restarted. (Ellen)
• Missed scans 054-0744 to 054-0821. (Ellen)
• 08:18 UT schedule resumed on new module: HOB+0102, first scan 054-0823. This makes the disc position about 2080 GB behind schedule. (Ellen)

Hobart 12m:

Disk VSN: HOB+1011

Data volume at beginning: 5093 GB

• Experiment started OK (Ross).
• 07:40 UT: pcfshb ran out of disk space. Deleted files in auto_cor_test older than 5 days. (Dave)

Katherine 12m:

Disk VSN: HOB+0124

Data volume at beginning: 8331 GB (missing 23 GB)

• Schedule not called from previous experiment, had to start manually. Missed scans 052-1901a through 052-1905 (Ross).

Yarragadee 12m:

Disk VSN: HOB+0121

Data volume at beginning: 15234 GB (missing 23 GB)

• Schedule not called from previous experiment, had to start manually. Missed scans 052-1901a through 052-1907a (Ross).

Hb

VSN: HOB+1011 ; 990.7 GB

• 19:00 UT - Experiment started OK (vk)
• RFI in last two channels (Lucia)

Ke

VSN: HOB+0124 ; 4198.3 GB

• 19:00 UT - Experiment started OK (vk)
• 21:35 UT - disk_pos ~20 GB in excess (vk)

Yg

VSN: HOB+0121 ; 11873.6 GB

• 19:02 UT - Halted schedule to check on mk5 error (vk)
• 19:09 UT - Asked John to power-cycle the mk5. After second round of power-cycle, mk5yg was cooperative (vk)
• 19:32 UT - First successful scan on 051-1932. Estimated deficit is 78.3 GB (vk)
• 20:35 UT - disk_pos ~10 GB in deficit (78.3 GB from prev. entry taken into account) (vk)

Hh

19:26 UT - It looks like HartRAO just joined the party (vk)

Hobart 12m:

Disk VSN: HAY-0055

Data volume at beginning: 1.4 GB (test recording + fringe check)

• 00:11 UT : disc pos ~40 GB behind - Rxmon stop command returns “operation not permitted; no process killed” not sure what to do since everyone is in the WHS training… (Ellen)
• 01:41 UT : Rxmon stopped and started. (Ellen)
• 02:10 UT : swapped modules over. Looked like everything was running again fine but then the mark 5 went crazy and tried to record to both modules. We did a full power cycle 3 times to fix this. Scans between 050-0211 and 050-0239 are either bad or were missed. (Ellen)
• Schedule resumed for scan 050-0241. (Ellen)
• 09:46UT: mark 5 lost connection. Halted schedule tried SSReset to no avail, hard rebooted as superuser, seemed to fix the problem. Schedule restarted 09:51, missed scans 050-0944b to 050-0956a (Imogen)
• 10:11UT: noticed that the time offset in the formatter (when issued mk5=dot? command) was more than 10ms. In fact, it was 50ms. However, this value fell with the next check and continued to fall until it was below 10ms. (Imogen)
• 10:54UT: mark 5 keeps losing connection, not sure why (Imogen)
• 11:49UT: scans 050-1146 to 050-1156 were lost due to attempt to heal the “ERROR m5 -104 mk5cn: time-out, connection closed” (restarted DIMino and fs, helped for now) (Liza)
• 12:02-04UT: DIMino stopped, restarted it twice, affected scans 050-1202, 050-1204a (Liza)
• 13:48-13:59UT: Problems with Mark5B, restarted DIMino repeatedly. Scans affected are 050-1348 to 050-1358. Meanwhile, clock jump occured, fmset synch performed 14:01;23 between scans 050-1400 and 050-1402 (Liza) Due to the Mark5 resetting, following scans were recorded to a wrong module UAO-0026/2000:
  • 1 a1508_hb_050-1356+
  • 2 a1508_hb_050-1358
  • 3 a1508_hb_050-1400
  • 4 a1508_hb_050-1402
• 16:45 restarted DIMino, fs, schedule. Scans 050-1645 and 050-1647 were affected (Liza)
• By the end, it recorded 300GB less due to repeated ERROR m5 -104 mk5cn: time-out, connection closed and attempts to recover it (Liza)

Katherine 12m:

Disk VSN: HOB+0124

Data volume at beginning: 1.12 (test recording + fringe check)

• 22:35 UT : clkoff returned a value ot ~E-001, while maserdelay was fine. “counter” didn't help, neither did “fmset” (Ellen)
• Scans 049-2235 to 049-2245 are bad.
• 22:47 UT : restarted the DBBC session and reconfigured, clkoff is still giving value of ~E-002. I open fmset again and the values all matched up, but I synced it any way in the hope of fixing the clkoff issue. But this actually un-synced the 3 numbers… “counter” didn't fix it this time either. (Ellen)
• 23:00 UT : Jamie walks in the room and suddenly the it decides to fix itself >.< (Ellen)
• 23:02 UT : Schedule wouldn't start properly, then the field system wouldn't respond to any commands. Tried to kill it but it wouldn't die properly so I couldn't restart it. Dave came to the rescue and killed it properly so I could restart the schedule at 23:21 UT. For most of this time the mark 5 was recording non-stop. Missed scans 049-2247a to 049-2326. (Ellen)
• Due to continuous recording, disc pos now ~53 GB ahead of schedule. (Ellen)
• Scan 049-2327 completed successfully. (Ellen)

Yarragadee 12m:

Disk VSN: HOB+0121

Data volume at beginning: 8369 GB

• We had some initial trouble with the module or mk5 when it refused to do a test record. The pointers were noticed to be at ~4TB rather than the expected ~8TB. The module eventually came good after:
  • moving the module to the other bank
  • rebooting mk5yg
  • running SSReset
  • restarting DIMino several times,
  • running “mk5=recover=2”
• The DBBC issue causing problems in S band is ongoing

• The wind monitor has not been able to connect from 22:00 UT to 00:30 UT (and counting). (Ellen)
• By the end, it recorded 100GB less (Liza)