User Tools

Site Tools


operations:handover

This wiki is not maintained! Do not use this when setting up AuScope experiments!

Handover notes

Please use this page to write up any notes on anything that needs to be passed on to the next observer or should go into the end of experiment message.

Notes should be placed below the horizontal line as a new blog entry.

Continuing problems should be copied to the Current Issues page.

**Template for Handover notes:**

Be sure to add any specific notes about the experiment if required, for example “Special back end configuration for x reason”. Put as much detail as you can - the more the better.

It is ESSENTIAL to put your name after EVERY entry. Copy and paste the template into the new handover notes

====== Experiment Name ======

**Hobart 12m**

Disk VSN: abc-123 | Data volume at beginning: 2.34 GB

   * 1300UT a comment from me (user)

**Katherine 12m**

Disk VSN: | Data volume at beginning:  GB

   * 

**Yarragadee 12m**

Disk VSN: | Data volume at beginning:  GB

   * 1300UT Experiment started okay (JMc).

Format for entering a comment It is important to stick to this formula - there is a good reason we need this information, so please include it!

Time stamp in UT * Comment * (Your Name)

For Example:

20:00UT: Wind stows, Missed scans 012-3456 to 012-3457 (Jim)

20:50UT: Mark5 lost connection, had to restart Mark5 and reconfigure DBBC, called Jamie on call to fix, missed scans 012-3456 to 012-4567 (Jim)

Again - as much detail as you can. It is important to remember that these notes will be sent to the correlator, so please make sure they make sense to others.


R4769

I think i entered 'schedule=r4769$s' at 18:26 UT and so it went to the second scan instead of the first.

hobart

Module: HOB+0125.

Disk_pos at beginning: 14159.002 GBs

  • Experiment started one scan late. Missed scan 343-1800 (Lucas).

Katherine

Module: HOB+0131.

Disk_pos at beginning: 7843.826 GBs

  • Experiment started one scan late. Missed scan 343-1800 (Lucas)

Yarragadee

Module: BKG+0052. NOTE : Yarragadee is not detecting the vsn however it still registers a disk present and seens to be recording to it.

Disk_pos at beginning: 0.000 GBs

  • Experiment started one scan late. Missed scan 343-1800 (Lucas)
  • 09:25UT eremote control showing blank in status monitor and logging and operator input windows. fs still running on pcfsyg. revived with ctrl shift e like with setup. (Liz)
2016/12/08 18:19 · Bryn Emptage · 0 Comments
aua015

pre-exp: Manually drudged snp, and proc files from the skd located in /home/oper/ on each machine – hopefully these are all fine

Hobart

Module: GSFC+022 (8TB), start pos 0.32 GB

  • 1800UT - Experiment started ok (JS)
  • 0243UT - wind stow. missed scans 343-0239 through 343-0258 inclusive (released shortly in between). (Lucia)
  • 0425UT - wind stow. Started in middle of scan 343-0415, through to 343-0430. (Lucia)
  • 0553UT - wind stow. missed scans 343-0554 through 345-0648
  • Previous wind stow was not recovered from. Had to reset drives etc however hb then immediately wind stowed again. Turns out it was really stuck and the technique on the current issues page didn't work after multiple times. However, I did a 'reboot system' on a UoT Antenna control (or reboot central, maybe both) and after I did the antenna=off, terminate, etc thing it suddenly worked. I'm guessing it needed a full system reset to clear it's queue (maybe?). (Lucas). First valid scan after prolonged wind stow event (see above) 345-0648
  • Missed scans due to an auto-stow event. No alarm at beginning of auto-stow but alarm when 'previous source in schedule not reached'. 4mins later auto-stow was released and restart after auto-stow was successful. Missed scans were 343-1231 and 343-1234 (Lucas).

Katherine

Module: HOB+0131 (32TB), start pos 0 GB

  • 1800UT - Experiment started ok (JS)

Yarragadee

Module: IAAE-018 (8TB), start pos 1606 GB

Module: JIVE-057 (2TB), start pos 0 GB

  • 1740UT Yg short on modules, will have to reuse IAAE-018 and spill-over onto the 2TB in bank A (JS)
  • 1800UT - Experiment started ok (JS)
  • 1952UT recorded data lagging expected by 7.24 GB (JS)
  • 2053UT recorded data lagging expected by 9.93 GB (JS)
  • 0503UT recorded data lagging expected by 28 GB (Lucia)
2016/12/07 18:13 · Bryn Emptage · 0 Comments
t2115

Hobart

Module: HOB+0125, start pos 12424.237 GB

  • 17:30 UT - Experiment started ok. (Ross)

Katherine

Module: UVLBI-29, start pos 0.000 GB

  • 17:30 UT - Experiment started ok. (Ross)

Yarragadee

Module: IAAE-018, start pos 0.000 GB

  • 17:30 UT - Experiment started ok. (Ross)
2016/12/06 17:17 · Bryn Emptage · 0 Comments
R1769

Fringe checks done with Hb, Ke, source 1921-293, only one fringe found (channel 04).

Hobart

Module: HOB+0125, start pos 10762 GB (Jamie said this module was good to record to, leaving the empty 8 TB)

  • During set up, FS reporting “-888 DBBC PPS error” every few minutes. Tried restarting + reconfiguring dbbc many times. Fmset reports no problems, autocorrelation of test scan looks fine. (Ellen)
    • Jamie fixed this by just running clkoff and maserdelay…
  • 17:00 UT - Experiment started ok. (Ellen)

Katherine

Module: BKG-0037, start pos 0 GB

  • 17:00 UT - Experiment started ok. (Ellen)

Yarragadee

Module: USN-0131, start pos 742 GB

  • During setup, everything worked fine until I tried to record a test scan, then the mark 5 froze and reported connection error. Restarting and rebooting and recycling the power made no difference; everything was fine until it tried recording every single time. (Ellen)
    • J-Mac determine that module HOB+0127 (7218 GB used out of 32 TB) is busted and so this is being shipped down to be fixed.
  • 17:00 UT - Experiment started ok. (Ellen)
2016/12/05 08:54 · Bryn Emptage · 0 Comments
ds338

Note that logs (on ops8) are under day number 337

All three telescopes:

  • 19:07:35 ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Bryn)

Hobart 12m:

Disk VSN: HOB+0125

Data volume at beginning: ~2TB

  • Experiment started OK (Jonny)
  • 2016.338.07:50:35 ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Nick)
  • 14:04:35 ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Bryn)
  • 07:09 UT ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Jonny)
  • 22:00 UT Experiment ended early due to a disruption with Hobart.

Previous source in this schedule not reached before new source was commanded for scans: 338-2224,

Katherine 12m:

Disk VSN: MAT+0005

Data volume at beginning: 0GB

  • Experiment started OK (Jonny)
  • 01:30UT Unable to resolve hostname auscope.utas.edu.au. Reissuing ./dysched.pl ds337 ke command into PCFs. Appears to be OK again.
  • 06:20:41UT ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Jonny)
  • 2016.338.07:04:38.15?ERROR fl -1 Previous source in this schedule not reached before new source was commanded (Nick)
  • 2016.338.07:50:34 ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Nick)
  • 06:48 UT ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Jonny)
  • 07:09 UT ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Jonny)
  • 08:58 UT ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Jonny)
  • 22:00 UT Experiment ended early due to a disruption with Hobart.

Previous source in this schedule not reached before new source was commanded for scans: 338-2202, 339-0248, 339-0521, 339-0543

Yarragadee 12m:

Disk VSN: HOB+0127

Data volume at beginning: 0GB

  • Experiment started OK (Jonny)
  • 01:10 UT, network connection dropped out. E-RemoteCtrl unavailable and can't VNC to PCFS. Can't access IP switch. Unable to use 'vncviewer yg-via-internode', suggesting this is perhaps more than just a VPN issue. Will monitor over the next 20-30 minutes. (Jonny)
  • 01:40UT Still unable to establish a connection. Unable to contact Yg - likely to be power issues
  • 06:50UT Jamie looking into the above
  • 07:00 UT things are back up and running. The dynamical scheduling appears to have stopped at approx 11:57 on pcfsyg due to being unable to resolve auscope.phys.utas.edu.au
  • I can't find the weather being logged (wtg/wx) (Nick)
  • 2016.338.07:04:38.15?ERROR fl -1 Previous source in this schedule not reached before new source was commanded (Nick)
  • 2016.338.07:50:34 ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Nick)
  • Consistently Yarragadee is recording 10-15% less per scan than the other 2 stations. (Ellen)
  • 05:12 UT - FS froze either due to the mark 5 crashing or causing it. Tried restarting both but it didn't working. Phone Jamie because the LEDs were flashing on the mark 5 module. (Ellen)
  • 05:40 UT - Jamie restarted the mark 5 and it decided to work. First can back 339-0548. (Ellen)
  • 07:09 UT ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Jonny)
  • 08:58 UT ALARM: error fl -1 previous source in this schedule not reached before new source was commanded (Jonny)
  • 22:00 UT Experiment ended early due to a disruption with Hobart.

Previous source in this schedule not reached before new source was commanded for scans: 339-0057a, 339-0204a

2016/12/03 00:14 · Bryn Emptage
HOB009

Hobart 12m (Hb)

  • 07:00 UT - First scan missed due to messing around with mark 5. Numerous hard and soft reboots done to get it to behave. Literally had a fit for no reason. (Ellen)
  • 03:55 UT - Alarm: Delay through DBBC has changes. Delaylive window shows a rapid change. DDBC was reset (Lucia).

Hobart 26m (Ho)

  • 07:00 UT - Experiment started ok. (Ellen)
  • 08:53 UT - Alarm reporting “previous source in this schedule not reached before new source was commanded” after stopping recording for scan 336-0853. Slewing during scan. No alarms following this. (Ellen)
  • 10:24 UT - Another alarm for slewing during scan 336-1023. (Ellen)
  • 11:21 UT - Slewing during scan 336-1120. (Ellen)
  • 23:50 UT - schedule halted for maintenance. Brett needed to cool the multifeed. I think the recorded kept running thorugh the schedule halt. now 302 GB too much. Adjusted disk offset in Log monitor accordingly. Scans lost 23:27-23:51. (Lucia)
  • 01:13 UT - Wind stow. Lost scans: 337-0058 through 337-0114 inclusive. (Lucia)
  • 01:30 UT - wind stow. Ongoing, schedule halted at 02:03 UT, resumed at 02:42 UT. Lost scans: 337-0131 through 337-0249. First good scan 337-0251. (Lucia)
  • 04:30 UT - schedule halted for antenna maintenance (Brett). Lost scans: 337-0416 through 337-0433 (Lucia)
2016/12/01 07:17 · Bryn Emptage · 0 Comments
/home/www/auscope/opswiki/data/pages/operations/handover.txt · Last modified: 2018/10/29 16:27 by Jesse Swan