====== Manual extraction and logging of Yarragadee weather data ======
We usually have a weather station connected to pcfsyg. However, if it is unavailable or broken, there is another weather sensor (used for the SLR) and it is possible to extract the data and merge it with the PCFS logs before they are submitted. Below is the procedure for doing this (a detailed description as well as a summary). Note that the external logs are updated every 30 minutes, so the Yg logs can only be processed 30 mins or more after the experiment finishes.

===== Using a script =====
On ops2 in ''/home/observer/ask/yglogs'' lives a script called ''yg_wx_logs.py'' which can be used to merge the logs with the weather data. To run the script type in 
<code> python yg_wx_logs.py </code>

The script will prompt the user for
  * Experiment name,
  * Date of the experiment,
  * Start time,
  * End time.

With the given information, the script will

  * Download the required mets files containing the weather data using sftp,
  * Download the log file from pcfyg,
  * Merge the log from pcfs with the downloaded weather data,
  * Backup the original log on pcfsyg with a ''_original2.log'',
  * Copy the modified log back to the pcfsyg.

Once all that is done you can run the flogit script to transfer that log to the IVS servers. As usual please eyeball the log to make sure the script has done the right thing.


----
It's probably best to modify the PCFS log on pcfsyg, then follow the usual end-of-experiment procedure. This way the log file will be the same on pcfsyg, ops2, and on the IVS servers. The procedure below uses the experiment r4750 as an example:

===== Detailed description =====

  - Identify the start date of the experiment. There's a separate log of weather for every UT day, so you'll need to retrieve the log for the start day as well as the following day. In the case of R4750, the start date was July 28 (day 210) at 18:30 UT, so we need logs from July 28 and 29.
  - Once the experiment is complete (and it's more than 30 minutes from the end), open a terminal on pcfsyg. Do this in the pcfsyg VNC session, especially if you want to plot the data, which is an option later.
  - <code>cd /usr2/log/wx2</code>
  - The weather data are on an sftp server at Yg. Use the usual observer password. The file naming format is ''mets_<yyyy>_<mm>_<dd>.dat'' so we want ''mets_2016_07_28.dat'' and ''mets_2016_07_29.dat'' <code>sftp -oPort=26 observer@59.167.111.242
cd metdata
dir
get mets_2016_07_28.dat
get mets_2016_07_29.dat
quit
</code>
  - The weather data files now need to be converted to the correct format for a PCFS log. There's a script to do that called ''metdat_format.sh''. By default it writes the output log to the screen, so send it to a temporary file. e.g. <code>metdat_format.sh mets_2016_07_28.dat mets_2016_07_29.dat > wx_tmp.log</code>Specify the input data files in chronological order.
  - Now you need to edit the new log file and delete any weather measurements before or after the experiment. There's a script for that too called ''metdat_startstop.sh''. Usage is <code>metdat_startstop.sh <input_file_name> <start_time> <end_time>
Where the time format is ddd.hh:mm. The output from this should be ready to be merged into the original log file, so give it a sensible output name and put it in the log directory (''/usr2/log''). </code> For example for R4750: <code>metdat_startstop.sh wx_tmp.log 210.18:30 211.18:30 > /usr2/log/wx_r4750yg.log</code>
  - You can now merge the original log file and the weather data. Change to the /usr2/log directory <code>cd /usr2/log</code>
  - Check that the weather data start and stop at the right time. Show the first 5 lines <code>head -5 wx_r4750yg.log</code> Show the last 5 lines <code>tail -5 wx_r4750yg.log</code> If something is amiss, you may need to go back to the start and work out what happened. You can also plot the weather data quite easily: <code>metdat_plot_weather.sh wx_r4750yg.log</code>
  - It's a good idea to make a backup of the original log first <code>cp r4750yg.log r4750yg_original.log</code>
  - Processing the log is a two step process. 
    - First, you want to remove any existing weather data from the log as it will be blank or contain errors <code>grep -v /wx/ r4750yg.log > r4750yg_no_wx.log</code>
    - Next merge the field system and weather logs to produce a final version:<code>sort -s -m -k 1,1.20 -o r4750yg.log r4750yg_no_wx.log wx_r4750yg.log</code>
  - Finally, make a few spot checks of the final log by eye to see if it looks OK. Hunt around for the wx lines and see if they look right and are in the right place.
  - You can now go back to ops2 and [[operations:documentation.ivs.logs|process the log file and submit a end email as usual]]. Please add a note to the email to say that weather data were merged into the log after the experiment.

===== Executive summary =====

This assumes R4750 is being processed.
  - Identify the start date of the experiment.
  - After the experiment AND 0 UT open a terminal in the pcfsyg VNC session.
  - <code>cd /usr2/log/wx2
sftp -oPort=26 observer@59.167.111.242
cd metdata
dir
get mets_2016_07_28.dat
get mets_2016_07_29.dat
quit
metdat_format.sh mets_2016_07_28.dat mets_2016_07_29.dat > wx_tmp.log
metdat_startstop.sh wx_tmp.log 210.18:30 211.18:30 > /usr2/log/wx_r4750yg.log
cd /usr2/log
head -5 wx_r4750yg.log
tail -5 wx_r4750yg.log
metdat_plot_weather.sh wx_r4750yg.log
cp r4750yg.log r4750yg_original.log
grep -v /wx/ r4750yg.log > r4750yg_no_wx.log
sort -s -m -k 1,1.20 -o r4750yg.log r4750yg_no_wx.log wx_r4750yg.log</code>
  - Make a few spot checks of the final log by eye
  - You can now go back to ops2 and [[operations:documentation.ivs.logs|process the log file and submit a end email as usual]]. Please add a note to the email to say that weather data were merged into the log after the experiment.