User Tools

Site Tools


operations:documentation.logmonitor_eremotectrl

This wiki is not maintained! Do not use this when setting up AuScope experiments!

This is an old revision of the document!


Starting the log monitor

For some background information:

  1. First make sure that e-remote control is running for the antenna you wish to monitor.
  2. In e-remote control, in the “Logging and Operator Input” section, click on the red dot icon (“record the logfile”):

    This will open a dialog window where you need to give a name for the log file to be saved. Select the “Browse for other folders” option, then double-click on “logs” at the end of the list on the left (this is a shortcut to /vlbobs/ivs/logs). Finally, give the log file a name at the top of the dialog. Suggested syntax is
    <experiment_name>_<station_code>_econtrol.log

    e.g.: a file is saved in this location

    /vlbobs/ivs/logs/r4456_yg_econtrol.log
  3. The red dot icon in econtrol should now show a green bar in the top left to indicate that the log is being recorded:
  4. Type a command into the Operator Input section (next to the blue triangle) so that the new log file is created. e.g.
    clkoff
  5. Now start the log monitor from the Applications menu on ops2:
    Applications → AuScope → Log monitor
  6. You will be asked which station is being monitored (type 1, 2 or 3) and then the GUI will start.
  7. Select
    File → Open Log File
    and type in the name of the log file you just created (e.g. /vlbobs/ivs/logs/r4456_yg_econtrol.log) or click on Browse and find the file.
  8. Click OK and the monitor should now be tracking the log file and checking for problems.
  9. At the moment, this program will
    1. monitor the difference in delays between maser and GPS and formatter and GPS, and sound an alarm if they differ by more than 1 us
    2. watch for all other error messages and either beep or sound an alarm depending on severity
    3. watch for Mark5 sync errors that require fmset to be run.
  10. You can acknowledge an alarm which will turn off the siren. You can also mute the alarm for 5 mins at a time.
  11. If you put in your mobile number and enable the SMS alarm, you'll get a text message whenever an alarm goes off.
    1. You can control the behaviour of SMS alarms remotely by putting comments in the PCFS log:
  12. There is a watchdog feature which remembers when the last field system log entry was received. If it is more than 15 minutes ago, the log monitor will start to beep. This is to trap field system software failures and network dropouts.
    1. This 15 minute timeout can be changed from the graphical interface.
    2. The log monitor won't start checking the time difference until the first new log entry is received
    3. Once a setup is complete there may be several hours before the next log entry at the experiment start. In this case, a series of incessant beeps after 15 minutes could get annoying. Pressing the “Reset watchdog” button will reset the log line counter to zero, meaning that the time difference will not be checked again until there's a new entry.

To set the optimal timeout values in the log monitor open a terminal on ops2 and type ./watchdog_check.sh <experiment name>, this will return a value to input into the log monitor for each telescope involved in the experiment. For example:

./watchdog_check.sh r4734

Will return:

hb12: Set watchdog timeout to 12 minutes
ke12: Set watchdog timeout to 16 minutes
yg12: Set watchdog timeout to 19 minutes

Note the log monitor script can also be started on ops6 (the PC in the “Lounge”). Just open a terminal window and repeat steps 5 and 6 above. Both ops2 and ops6 share the /vlbobs directory, so log recording needs to be started only from one place.

Configuring the log monitor

The log monitor program reads a configuration file which tells it what to do when it sees an error. It will either beep, ring the alarm or ignore the error. The file is on ops2 in /usr2/control.xx/alarm_actions.txt where xx is the two-letter code for the antenna. To change how the monitor responds to alarms:

  1. Edit the file
  2. look for the alarm code you want to change the response to and change it (can be BEEP, ALARM or IGNORE)
  3. Close the file and re-start the log monitor

For example, if you want to stop the mark5 error 900 from ringing the alarm to just beeping, edit the file and change this line

M5 -900 ALARM

to this

M5 -900 BEEP
/home/www/auscope/opswiki/data/attic/operations/documentation.logmonitor_eremotectrl.1463630264.txt.gz · Last modified: 2016/05/19 03:57 by Tiege McCarthy