The correct monitoring tools were not set up on ops8 when I started (no eremote control, no log monitors or delaylive's)(Lucas)

Yarragadee 12m

Disk VSN: HOB+0056 | Data volume at beginning: 0.0 GB

1. 1730UT Experiment started OK (Jay).
2. 6:00UT Very large positive(!?) data difference. +200TB or about 5%. Module is large enough to handle additional data (Lucas).

Hobart 12m

Disk VSN: flexbuffhb | Data volume at beginning: 0.0 GB

1. 1730UT Experiment started okay (Jay).
2. 7:00UT When I set up the delaylive, noticed a series of clock-jumps on the dbbcho starting at 5:00UT, then again at 6:00,6:30 and 7:00. Clock jumps are on the order of -0.015us. (Lucas)
   1. Have not reset the dbbcho (yet) as the summary script reports okay delays and samplers. Appears stable at the moment but if it jumps again I will reset (Lucas)
   2. Jumped again at 8:00 UT, halted and reconfigured dbbcho. Missed scan 057-0821b, 0825.
   3. Called Jamie. clkoff isn't actually indicative for dbbcho problems when hooked up to Hobart12m. So don't worry about delaylive hb.

Katherine 12m

Disk VSN: flexbuffke | Data volume at beginning: 0.0 GB

1. 1730UT Experiment started okay (Jay).
2. Drives lost connection, got stuck. Required a thorough reboot (reboot drives didn't work), reboot system then a 2min wait+[drives on]+operate+source=disable+antenna=open+antenna=operate worked (Lucas).
   1. Missed scans 057-0839 up to 057-0851.
3. Flexbuffke got into a nasty state and wasn't happy. The first sign was the data rate on Grafana went from the 'normal' smooth 3Gbps scans to ugly, spiky and somewhat random data rates. Another sign was the number of dropped packets in “mk5=evlbi?” – maybe we should add this to katherine midob?
   1. The current fix is to (halt obviously), umount all the drives, remount them, restart jive5ab and re-setupsx.
      1. On flexbuffke there are two scripts in /tmp/. After logging in as su (ssh observer@flexbuffke then su in random terminal), run script /tmp/UMountAllFlexbuff.sh; wait make sure they all unmount, then run /tmp/MountAllFlexbuff.sh. After this you need to go to the flexbuffke jive5ab window (normally running on the vnc, with lots of “Reply: !”–type stuff) and ctl-C OR just pkill jive5ab from observer@flexbuffke. Then restart by up-arrowing or typing “/opt/jive5ab-2.9.0 -m2 &” or maybe …“-3.0.0” jury is out which one atm. Then setupsx in the field system and make sure the datarate is okay and “mk5=evlbi?” looks happy and is increasing (Lucas).