User Tools
Sidebar
This wiki is not maintained! Do not use this when setting up AuScope experiments!
Handover notes
Please use this page to write up any notes on anything that needs to be passed on to the next observer or should go into the end of experiment message.
Notes should be placed below the horizontal line as a new blog entry.
Continuing problems should be copied to the Current Issues page.
**Template for Handover notes:**
Be sure to add any specific notes about the experiment if required, for example “Special back end configuration for x reason”. Put as much detail as you can - the more the better.
====== Experiment Name ====== **Hobart 12m** Disk VSN: abc-123 | Data volume at beginning: 2.34 GB * 1300UT a comment from me (user) **Katherine 12m** Disk VSN: | Data volume at beginning: GB * **Yarragadee 12m** Disk VSN: | Data volume at beginning: GB * 1300UT Experiment started okay (JMc).
Format for entering a comment It is important to stick to this formula - there is a good reason we need this information, so please include it!
Time stamp in UT * Comment * (Your Name)
For Example:
20:00UT: Wind stows, Missed scans 012-3456 to 012-3457 (Jim)
20:50UT: Mark5 lost connection, had to restart Mark5 and reconfigure DBBC, called Jamie on call to fix, missed scans 012-3456 to 012-4567 (Jim)
Again - as much detail as you can. It is important to remember that these notes will be sent to the correlator, so please make sure they make sense to others.
r1758
Hobart 12m:
Disk VSN: HOB+0120
Data volume at beginning: 23293.563 GB
- 17:00UT Experiment started OK. (AK)
- 00:09UT Paused to swap modules. Back observing with scan 264-025. Scans missed 264-0009b to 264-0020 inclusive (Jim)
Katherine 12m:
Disk VSN: BKG+0141
Data volume at beginning: 6.090 GB
- During setup, ke dbbc needed a power cycle, This fixed the issue of drifting time difference and bad auto correl spectra. (AK)
- 17:00UT Experiment started OK. (AK)
- 17:11UT had the following error “error s5 -104 rfpcn: time-out, connection closed” Restarted rxmon and ifbox. First clean scan without this error was 17:45, took me a while to realise I need su not sudo to login as super user (sleep deprived :P ) (AK)
- 20:58UT reconfigured the dbbc, between scans so no data should be affected. (AK)
- 21:00UT had to sync fmset, halted the schedule. Missed scans 263-2102 and 263-2104b . Reconfiguring the dbbc and synccing the fmset seems to have fixed the bad channel in auto correlation spectra and returned the tsys values. (AK)
- 23:51UT Halting to reconfigure DBBC. Scans lost from 263-23 to 264-002a inclusive (Jim)
Yarragadee 12m:
Disk VSN: GSFC+017
Data volume at beginning: 1.987 GB
- 17:00UT Experiment started OK. (AK)
hob004
General: 06:50UT UTAS Sandy Bay struck by lightning, power either went out during strike or switched to backup (Ross). 06:53UT Second strike knocked power out again (Ross).
Hobart 12m:
Disk VSN: HOB+0120
Data volume at beginning: 21130.8
- 19:00UT Experiment started okay. (AK)
- 10:20UT - ERROR st -27 Computer ACU time difference exceeded 0.25 seconds continuously experienced for 1 minute then fixed itself. Probably a network issue. (Ellen)
- 11:30UT - Since doing the last checklist the delay difference seems to be slowly drifting although it seems to have levelled out for the moment. Will keep a close eye on it. (Ellen)
Hobart 26m:
Disk VSN:HOB+1010
Data volume at beginning:0
- 19:00UT Experiment started okay. (AK)
- 04:22UT Hobart has been recording less data than scheduled throughout the experiment at an average rate of 17GB per hour (Ross).
- 09:57UT - Disc pos ~240 GB behind (is supposed to be close to 1400 GB total). Jamie has fixed something so that the scans are losing 1s at the beginning instead of 5-6s. So it should lose data more slowly now. (Ellen)
* 11:30UT - Disc pos is now 250 GB behind so appears to be losing data more slowly now. (Ellen)
R4757
Hobart
Power levels on last 2 channels of iread (s-band) keep fluctuating and can't stay near target for long. Jim thinks it's probably RFI so we can't do anything about it at the moment. Brett should be able to investigate when he gets in tomorrow. (Ellen)
Module: HOB+0120, start pos 19384.631 GB
- 18:30 UT - Experiment started ok. (Ellen)
Katherine
Dimino wouldn't run properly. Rebooted mark 5. It wouldn't let us ssh in but we could ping the server. Jim eventually got it started again. (Ellen)
Module: IPA-0022, start pos 0 GB
- 18:30 UT - Experiment started ok. (Ellen)
- 20:45 UT - All tsys values overflowing for about 15 minutes. There is no storm to interfere and this stretches across multiple scans so is not position dependent. Continuing to watch. (Ellen)
- 22:00 UT - Tsys values are still all overflowing, Jim thinks there's nothing wrong with the data just the values it's displaying. He will investigate at 9am. (Ellen)
- 22:30 UT - Every few scans the autocorrelation shows a jump in band 5 - noted in log at specific times. (Ellen)
- 16:15 UT - Channel 4 of the autocorrelation looked like a “step function” and all the system temps had disappeared. Reconfigured the dbbc between the 16:15 and 16:20 scans. No scan should be affected, auto correl spectra back to normal. Also updated the wiki page on this. (AK)
Yarragadee
Module: NTO-0007, start pos 0 GB
- 18:30 UT - Experiment started ok. (Ellen)
DYSCHD258
Hobart 12m
vsn HOB+0120, disk_pos 15524.45 GB
- 23:10 Setup and ready to run (JS)
- 0107UT Killed observing and disconnected from dynamic session so Brett could work on telescope
- 0540UT Resumed observing without issue
Katherine
vsn WSRT-061, disk_pos 0 GB Only a 2TB module, the recorder will switch to Bank B for another 2TB
- 23:10 Setup and ready to run (JS)
- Continuous errors at the start if a new scan(?) `error m5 -900 not while recording or playing`, appears benign likely just a command issued before disk_record=off (JS)
- 0357UT Noticed that FS at Ke wasn't responding. It hadn't crashed, somehow it was just (silently) unresponsive. Watch dog hadn't picked this up! I had to force kill fs from terminal (JS)
- 0401UT Ke removed, FS restart (JS)
- 0507UT Ke added back in, observing resumed (JS)
- 0800UT stopped responding, 20 minutes later it set the watchdog off and I eventually found that the field system wasn't responding. It had frozen midway through recording a scan so there is a huge amount (~700 GB) of useless data. Restarted everything and began dynamic observing at 0911UT. (Ellen)
- 2241UT Ke Disks also full, went to halt and remove from dynamic sched but someone beat me to it (2239UT) (JS)
Yarragadee
vsn USN-0138, disk_pos 0 GB 4TB module, should be enough
- 23:10 Setup and ready to run (JS)
- 1919UT Yg bank A full, no hdd in bank B
- yg halted and removed from dynamic sched (JS)
RD1609
Hobart 26m
Disk position 12779.582 GB, vsn HOB+0108 (16000 total space).
- 17:00:00 UT Experiment Started Ok (Lucas).
- 21:42 UT, missed scan 257-2339b. ERROR fl -1 Previous source in this schedule not reached before new source was commanded. (Lucas)
R1757
Hobart 12m
Data Recorded to MED-0038, disk_pos 0 GBs
- 17:00 UT - Experiment started OK. (Tiege)
- 18:47 UT - Alarm due to delay through DBBC changing, will monitor to see if the delay stabilises. (Tiege)
- 18:50 UT - Delay immediately stabilised at ~0.4 us, with variation from 0.398 us through to 0.403 us. Going to leave it be as long as it does not start drifting again. (Tiege)
- 19:26 UT - delay difference drifted again to 0.435 us, going to try and see if restarting DBBC software helps, seems to be staying at the nominal ~0.257 us post-reset. Halted schedule during 256-1925a, first scan after reset will be 256-1944. (Tiege)
- 22:59 UT - Have noticed that in the time between the previous two checklists ( time where I was asleep) the delay difference has very slowly drifted from ~0.26 us to 0.285 us, gradual enough that is hasn't yet set off the alarm. Jim will be taking over soon so I will mention it to him. (Tiege)
- 23:25 UT: DBBC was still causing a drift so rebooted the machine. Jim
- 23:37 UT: Restarted with scan 256-2347. OK now. Jim
Katherine
Data Recorded to USN-0069 (mk5ke) and USN-0017 (mk5-2ke), disk_pos 0 GBs
- 17:00 UT - Experiment started OK (Tiege)
- ~17:30 UT - Noticed that I was consistently getting the warning “WARNING: error sc -13 setcl: formatter to fs time difference 0.5 seconds or greater”. Tried fixing this in all the ways listed on the wiki. mk5=dot? returned large numbers for the last value. Monitoring fmset, the times seemed to be synced when data was not being recorded, however, when recording started mk5 became heavily out of sync. After trying multiple times to fmset, and to restart dimino on mk5ke, I attempted to power cycle the mk5ke using the ip switch. Unfortunately after the power cycle, I have been unable to ssh into mk5ke. As a work-around (hopefully), I have switched over to mk5-2ke for this experiment by terminating the fs, editing /usr2/control/mk5ad.ctl so that it runs of mk5-2ke and then restarting the experiment. (Tiege)
- 18:10 UT - Experiment seems to be running fine on mk5-2ke, however, preob clkoff command is failing as it should be calling clkoff-2, I am unsure what file too edit in order to adjust this, will attempt to find it now. (Tiege)
- 18:30 UT - Unable to find where to adjust clkoff to clkoff-2 in the preob, I know i need to edit the procedure files for midob and preob but I am not sure how. going to manually check delay difference until Jim starts in the morning and can fix. (Tiege)
- 18:25 UT - Total missed/dodgy scans list: 256-1700b to 256-1733b inclusive were recorded onto USN-0069 using mk5ke (potentially dodgy), scans 256-1742 through to 256-1802 were missed while switching to mk5-2ke, 256-1808 and 256-1809 WERE RECORDED on the new disk (USN-0017), then 256-1814 through to 256-1827b were missed attempting to correct clkoff issue. (Tiege)
- 18:30 UT - Experiment should be running as normal from scan 256-1833b onwards. Just note that 256-1808 and 256-1809 were both recorded to this module also. (Tiege)
- 00:58 UT - Observations stopped from 00:55:48 to 00:58:10 to put in another module in bank A. Jim
- 0720UT to 1030UT (JS)
- MK5-2KE crashed as well, this was a UPS issure but in the meantime MK5 was fixed
- GPIB fixed, clkoff/maserdelay now working
- Module USN-0017 transferred from mk5-2ke to mk5 bank A, recording will continue on this
- mk5 and dbbc reset, and synced with fmset
- First new scan 257-1039. Missed scans 257-0720b through 257-1035c (215GB deficit)
Yarragadee
Data Recorded to USN-0020, disk_pos 52.1 GBs
- 17:00 UT - Experiment started OK (Tiege)
