Current Issues

Symptom

• VNC for maser pc is not working from the drop-down menu
• Manually attempting to VNC to the maser pc returns some form of unreachable message

Solution

• Contact staff at Yg and ask them if they can reboot the maser pc
• Once the pc comes back online, it should now be reachable by VNC
• Start maser server software, top right corner on Desktop
• Start maser management software (and check everything is still nice and green)

The PIC in the IF rack at Yg has a problem that prevents diode switching and hence Tsys measurements. The preob procedure in station.prc has been edited to comment out systemp12. Will need re-enabling when tsys is back (Jim)

If the network connection to Yarragadee dies, the VPN could need reseting:

• How to reset the Yarragadee VPN

The weather sensor at Yarragadee is currently away for repair (humidity sensor is broken). Weather is still being logged but from a separate sensor that is not connected to the Field System. Until our sensor is repaired and returned, there is an extra procedure to go through to recover the weather information:

• Manual extraction and logging of Yg weather data

(added 14th Oct 2015, updated 9 Aug 2016)

Symptom

• Log monitor Hb alarms saying that no pcfshb activity for watchdog time limit.
• vncviewer pcfshb shows that pcfs is running, but there is a recurring error along the lines of: “wrong length written, file probably too large can't write to disk”
• pcfs out of disk space; disk full

Solution

• Open a terminal on pcfshb
• Enter command: “crontab -l”
• Two lines should be printed out, containing the string:“find /home/oper/auto_cor_test/”
• Copy the first line, starting from the “find” command to the end, and paste into the command prompt.
• Change the “+15” to “+10” and hit “Enter”
• Do the same for the second line of the crontab printout

Alternative Solution

• Open a terminal on pcfs[hb|ke|yg]
• In the terminal window type the following to see all auto-correlation files:

  ls /home/oper/auto_cor_test/

• Then type the following to remove the largest (unnecessary) file:

  rm /home/oper/auto_cor_test/*.epi
  rm /home/oper/auto_cor_test/*.png

Please note here any issues you have with the new eRC. Whenever there is a problem, it would be helpful for Alexander to know whether the time is still running in the eRC Status Monitor and what is your actual user name in the eRC (oper, oper(1),…).

• …

During set up of R1569, mk5hb's response to ntpq -np indicated very high offsets and high jitter (25 ms). Problem persists after a reboot. Not critical at the moment but needs watching in case it gets worse.

Antenna “stuck” as elevation brakes will not release. fs, halt schedule. Need local person to switch off big yellow rotatry power switch in pedestal, open power cabinet and switch to maintenance mode. Switch power back on, but this time to hand box control. Should be able to drive by hand. Switch power back over to remote mode. EM reset orange button press twice. HMI interface should now drive telescope might need to “Reboot Central” in HMI. fs, antenna=open, antenna=operate, resume schedule. Good luck with all of that….

………… I have had calls to assist. These seem to come more often during the change from one experiment to another. I have a suspiscion that if the change over period from one experiment to the next is brief, then the problem happens. Hope that helps. It particularly happens with Austral weekend experiments, or so it seems. Regards, Brett.

…………. It played up today again from around 8am Hobart time. Jack was able to move it in maintenance mode though. At midday Hobart I was able to drive the antenna for a while in remote after doing “reset timer”. But this only lasted about 5 minutes. Better get Mark Godwin to check it. Regards, Brett. 29/8/14

If the ops2 display locks up, or has odd sized displays, etc, first try powering off the screens - use the power pointi rather than the button on the controller. A switched power board (labelled) has been added to make this easier - it's under the screens on the left hand size and (badly) labelled as “screen power”. Once the screens are power cycled, you may need to run use “ctrl-alt-f1” and then “ctrl-alt-f7” to swap to the terminal and then back to the X-windows desktop.

• Note: Need to add notes on starting wind data monitoring for System Monitor. As root:

  /etc/init.d/Monica.weather start

  on pcfshb (Jim)

• Bug in log monitor for 26m: doesn't calculate length of last scan correctly (difference between Mark5A and 5B+ disk_pos readback)
• Document procedure for recovering from wind stow when it doesn't work automatically.
• What happened at Ke on Wed morning?
• Drudge remaining schedules. Schedules to C1407 now done.
• Make sure dimino is started via VNC session on pcfs
• Note schedules may finish early, check last stop time
• Checklist should have Station ID at top

If “mk5=dot?” gives a time difference (string of numbers at the end: -0.003224s ;) of ~0 or ~integer steps, then it is not necessary to run fmset, as the correlator can resolve this time step. If it is some other fraction between, then fmset needs to be run with either '+' or '-' to sync.

The S/X cryogenics have been warming up regularly. Please keep an eye on them and let Brett know ASAP if you notice any problems. Things to look out for are documented on the 26m Checklist page and the Log Monitor picks up on some of them.

By default, Log Monitor will ring an alarm if any of these conditions occur:

1. The temperature of the 70K stage exceeds 75K
2. The Dewar vacuum pressure exceeds 3.5 microns of Mercury

If either of these alarms go off, please phone Brett.

When the cryogenics are recovering (or while the problem is known but not yet fixed), you can change the alarm to a beep or silence it altogether. Do this from the Log Monitor menu: “Configure → Cryo monitoring” but please remember to turn the alarm back on once the receiver is working within range again.

Lastly, there’s one other thing to monitor which Log Monitor can’t ring an alarm on at the moment. The Helium supply pressure should stay above 275 psi. If it get’s below that, Brett needs to know ASAP. This quantity should be monitored via MONICA as part of the regular 2-hourly checks.

Occasionally the 26m drives will trip out with an encoder error. This should cause the log monitor to beep but doesn't ring an alarm (yet!). Example log message:

2014.127.05:29:38.24#antcn#Error: Antenna reported error from last command (0)
2014.127.05:29:38.24#antcn#Antenna control system Local control asserted by DESK
2014.127.05:29:38.24#antcn#Check if antenna is on-source
2014.127.05:29:38.64#antcn#Antenna drives not on, switching them on
2014.127.05:29:59.84#antcn#Successfully sent new source position to antenna

The antenna should recover the next time it is sent to a source or when there's an insource command issued. However, if you notice it happening, you can get things going again sooner with the command

onsource

Something has gone wrong with the position measurement setup and the readings from 'rxp' currently do not correspond to the receiver's actual position. Brett manually positioned the platform on 15/5, and the S/X receiver should be on axis.

We've noticed at Hb that it doesn't recover properly from wind stows. Check the wind speed (average should be below ~50 km/h) and the anemometer display on time[hb|ke|yg]. If it shows the antenna NOT in a stow state but the field system or eRemoteCtrl report it's in a stow state, then it needs fixing. Recovering seems to be problematical but this seems to work if nothing else does:

• On the pcfshb VNC session:

  • halt

  • disk_record=off

    (just to make sure the recorder is stopped)

  • antenna=off

    (should turn the drives off, check HMI on timehb)

  • terminate

    (This will exit the field system software)

• Then re-start the field system in a terminal with
  

  fs

  . Then in eRemoteCtrl:

  • proc=r1653hb

  • setupsx

• Then:

  • antenna=open

  • antenna=operate

    (check HMI and see if drives power up)

  • schedule=r1653hb

    (schedule should start. Wait for next source command and see if antenna slews)

Target power levels are listed on the DBBC IF Target Power Levels page:

If the power levels in the dbbc are not reaching their targets, check if the automatic gain control (AGC) is reaching its limit:

dbbcifa

dbbc=1,agc,1,45000,63,48147

The important part here are the last three numbers, 45000 here is the target power, 63 is the automatic attenuation, and 48147 is the measured power.

63 is the maximum attenuation agc can give you and our power is still too high so we need to add more attenuation to the IF chain.

The RF and IF Configuration page has with instructions on how to set attenuation manually.

If the isdx settings when drudging are wrong, for example:

 ifa=1,agc,1
 ifb=2,agc,4
 ifc=3,agc,2
 ifd=4,agc,2

Or if the response from iread looks like (for example)

2013.018.01:46:35.31/ifa/1,agc,1,1,63,28940
2013.018.01:46:35.31/ifb/4,agc,4,32000,63,32352
2013.018.01:46:35.31/ifc/2,agc,2,1,63,11389
2013.018.01:46:35.31/ifd/2,agc,2,40000,41,39524

this indicates that the DBBC target power levels are incorrectly set. To correct this, use pfmed and edit the ifdsx procedure to set the target levels directly in the current active schedule procedure file: e.g: r1593ke on pcfske

pfmed
pfmed: pf,r1593ke
pfmed: ed,ifdsx

ifa=1,agc,1,35000
ifb=4,agc,4,28000
ifc=2,agc,2,28000
ifd=2,agc,2,40000

pfmed: exit

Note that for Hobart26, the settings will be different (due to the LO frequencies of 8080/2020 MHz rather than 7600/1900 for the 12ms). e.g

pfmed
pfmed: pf,r1593ke
pfmed: ed,ifdsx

ifa=1,agc,2,40000
ifb=4,agc,1,48000
ifc=2,agc,2,32500
ifd=2,agc,2,35000

pfmed: exit

If the DBBC computer hangs or crashes and the vnc session and ssh server are unreachable you may need to reboot the machine. Details for how to do this can be found on the Internet Power Switch (IPS) Page

If you are unable to convince the DBBC, Mark5, and field system to sync their clocks, first follow some basic troubleshooting. Check the maser is working correctly, the gps clock (tac32) has the correct time on the (ho/ke/yg)time vnc, that ntpq -np returns a low offset and/or reboot the dbbc/Mark 5 and run fmset.

If all this fails you may need to reconfigure the dbbc into an 8 MHz (non CRDS) mode to get it to sync and then swap it back to 4.

Symptoms: It's not possible to set a VSN code. On the camera, both modules are shown as 'active' (i.e. red light).

Cause: This happens when both modules have a blank VSN. In this case, the Mark5 thinks it is in dual-bank mode (recording to two modules simultaneously) and the Dimino software in unable to recover from this.

Solution: There is an alternative to Dimino called jive5ab which allows relabelling of VSNs when in this dual-bank mode.

1. Log on to the Mark5 unit as oper and stop Dimino:

   EndDIM

2. Now start jive5ab:

   /opt/jivemark5a

3. You should now be able to send commands to the Mark5 as usual from eRemoteCtrl or the Operator Input window on the Field System VNC session. Go through the usual VSN setting procedure.

When you have completed the operations with jive5ab, you should kill the process before restarting DIMino. You can do this either by using Ctrl-C in the window running jive5ab or with the “kill” command - check the process number with “ps -ef | grep jive5”. Either way, allow a few seconds to pass between ending jive5ab and starting DIMino to let the diskpack controller settle.

If the Katherine skycam doesn't show up on the “web cameras” webpage, it could either be that VLC (which saves the images from the camera) or the FTP script, which uploads the image) has stopped working. To reset these, VNC to timeke:

1. If there is a black terminal window with text about FTP, close it. It runs every ~5 min, so there's no need to reopen it.
2. Close VLC (with the orange traffic cone icon). The is a shortcut to open it again on the Desktop (VLC SkyCam grabber).
3. To upload the latest image manually, you can run the FTP script manually by opening the ftp_script program in C:\thumbs

In the case of Yarragadee and Hobart: the script is ASC Uploader on the Desktop

In case of dodgy screen display at the Observatory (4 screens) open a terminal on ops1hb and type

observer@ops1hb:~$ ./ScreenFix.sh