Chair: Earl
Next meeting chair: Pradyumna
Attended: Jamie, Simin, Lim, Mas, Pradyumna, Jay
Apologies:
General Issues
auscope0 link vs new auscope site, (up and down with testing).
email exploder issues.
resume SI experiments from 16/4/21 - best time at end of r4 experiments, weekly, extra 1 hour, same mode.
Hobart 26 still out – hopefully back by easter's end
Handover Issues
started getting mk5 error 'error m5 -900 : not doing record'. find out that data was not being recorded. restarted dbbc3ke, did setup again, but didn't help. Warren looked deep into it and found out proc file had setupvgos commented out. After correcting that, recording started smoothly. I've faced the same issue with previous mix mode experiment. I think 'setupvgos' is commented out in 'si.prc', where we copy it for a particular experiment. Not sure whether it is a permanent thing or someone did some editing recently. Have to check further on this. (Jay)
si.prc has been fixed, setupsx procedure alias for setupvgos had been commented out (Warren)
Confirm safety of this fix, could cause clock breaks, should only run vgos at setup or on dbbc reconfig – flexbuff time out –
First good scan is from 1732UT (Jay)
0827UT ALARM: error st -998 reading systemclock1; ALARM: error st -999 illegal state error; Antenna communication software (antcn) problem; Try sending source=disable or antenna=open to reinitialise connection. “antenna=open” and “antenna=operate” brought back the connection after several times trying. Affected scans: 082-0828 to 082-0842a (Simin)
1830UT, halted the schedule. Sampler 3 and 4 are bad. Reconfigured the DBBC3. Resume schedule at 1919UT. Missed scans from 076-1832 through 076-1943 (mas)
2157UT halted the schedule again - DBBC3 delays bad again (JMc)
2208UT - Resumed. aua073 data deleted from flexbuffhb to free ~20 TB (JMc)
2210UT - DBBC3 failed again (JMc). restarting server program on dbbc3hb (JMc)
2322UT - Resumed after a full power cycle. Apparently ok… first good data expected from 2328 (JMc)
remote power cycle wiki notes
checkmk5 script not copied when sched etc. run on ops8… scp within bash not transferring…
notes on setup scan streams - sxtimetest cmd - create equivalent of checkmk5…
1800UT Did not recorded data in 'sxy' stream till this point. fs wasn't talking to dbbcho. restarted dbbcho, did a quick re-setup. first scan with all 3 stream is from 1802UT(Jay & Warren)
1908 UT /mnt/disk13 i/o errors (this was the one that flaked out last week) need to clone, replace. also means lost 3.2 TB free space. disk10 full (Warren)