Table of Contents

Manual extraction and logging of Yarragadee weather data

We usually have a weather station connected to pcfsyg. However, if it is unavailable or broken, there is another weather sensor (used for the SLR) and it is possible to extract the data and merge it with the PCFS logs before they are submitted. Below is the procedure for doing this (a detailed description as well as a summary). Note that the external logs are updated every 30 minutes, so the Yg logs can only be processed 30 mins or more after the experiment finishes.

Using a script

On ops2 in /home/observer/ask/yglogs lives a script called yg_wx_logs.py which can be used to merge the logs with the weather data. To run the script type in 

 python yg_wx_logs.py

The script will prompt the user for

With the given information, the script will

Once all that is done you can run the flogit script to transfer that log to the IVS servers. As usual please eyeball the log to make sure the script has done the right thing.

It's probably best to modify the PCFS log on pcfsyg, then follow the usual end-of-experiment procedure. This way the log file will be the same on pcfsyg, ops2, and on the IVS servers. The procedure below uses the experiment r4750 as an example:

Detailed description

  1. Identify the start date of the experiment. There's a separate log of weather for every UT day, so you'll need to retrieve the log for the start day as well as the following day. In the case of R4750, the start date was July 28 (day 210) at 18:30 UT, so we need logs from July 28 and 29.
  2. Once the experiment is complete (and it's more than 30 minutes from the end), open a terminal on pcfsyg. Do this in the pcfsyg VNC session, especially if you want to plot the data, which is an option later.
  3. cd /usr2/log/wx2
  4. The weather data are on an sftp server at Yg. Use the usual observer password. The file naming format is mets_<yyyy>_<mm>_<dd>.dat so we want mets_2016_07_28.dat and mets_2016_07_29.dat 
    sftp -oPort=26 observer@59.167.111.242
cd metdata
dir
get mets_2016_07_28.dat
get mets_2016_07_29.dat
quit
  5. The weather data files now need to be converted to the correct format for a PCFS log. There's a script to do that called metdat_format.sh. By default it writes the output log to the screen, so send it to a temporary file. e.g. 
    metdat_format.sh mets_2016_07_28.dat mets_2016_07_29.dat > wx_tmp.log

    Specify the input data files in chronological order.

  6. Now you need to edit the new log file and delete any weather measurements before or after the experiment. There's a script for that too called metdat_startstop.sh. Usage is 
    metdat_startstop.sh <input_file_name> <start_time> <end_time>
Where the time format is ddd.hh:mm. The output from this should be ready to be merged into the original log file, so give it a sensible output name and put it in the log directory (''/usr2/log'').

    For example for R4750: 

    metdat_startstop.sh wx_tmp.log 210.18:30 211.18:30 > /usr2/log/wx_r4750yg.log
  7. You can now merge the original log file and the weather data. Change to the /usr2/log directory 
    cd /usr2/log
  8. Check that the weather data start and stop at the right time. Show the first 5 lines 
    head -5 wx_r4750yg.log

    Show the last 5 lines 

    tail -5 wx_r4750yg.log

    If something is amiss, you may need to go back to the start and work out what happened. You can also plot the weather data quite easily: 

    metdat_plot_weather.sh wx_r4750yg.log
  9. It's a good idea to make a backup of the original log first 
    cp r4750yg.log r4750yg_original.log
  10. Processing the log is a two step process.
    1. First, you want to remove any existing weather data from the log as it will be blank or contain errors 
      grep -v /wx/ r4750yg.log > r4750yg_no_wx.log
    2. Next merge the field system and weather logs to produce a final version:
      sort -s -m -k 1,1.20 -o r4750yg.log r4750yg_no_wx.log wx_r4750yg.log
  11. Finally, make a few spot checks of the final log by eye to see if it looks OK. Hunt around for the wx lines and see if they look right and are in the right place.
  12. You can now go back to ops2 and process the log file and submit a end email as usual. Please add a note to the email to say that weather data were merged into the log after the experiment.

Executive summary

This assumes R4750 is being processed.

  1. Identify the start date of the experiment.
  2. After the experiment AND 0 UT open a terminal in the pcfsyg VNC session.
  3. cd /usr2/log/wx2
sftp -oPort=26 observer@59.167.111.242
cd metdata
dir
get mets_2016_07_28.dat
get mets_2016_07_29.dat
quit
metdat_format.sh mets_2016_07_28.dat mets_2016_07_29.dat > wx_tmp.log
metdat_startstop.sh wx_tmp.log 210.18:30 211.18:30 > /usr2/log/wx_r4750yg.log
cd /usr2/log
head -5 wx_r4750yg.log
tail -5 wx_r4750yg.log
metdat_plot_weather.sh wx_r4750yg.log
cp r4750yg.log r4750yg_original.log
grep -v /wx/ r4750yg.log > r4750yg_no_wx.log
sort -s -m -k 1,1.20 -o r4750yg.log r4750yg_no_wx.log wx_r4750yg.log
  4. Make a few spot checks of the final log by eye
  5. You can now go back to ops2 and process the log file and submit a end email as usual. Please add a note to the email to say that weather data were merged into the log after the experiment.