SeaExplorer Data Processing

[Kourtney-Burger]

Hello 👋

I work with the NOAA SWFSC ESD glider lab and we recently completed a collaborative project with Alseamar that included a "data as a service" contract for their passive acoustic monitoring SeaExplorer glider. We received summarized data (flat sci and eng timeseries files), however, I would like to reprocess the gli and pld files using PyGlider. We primarily use slocums with the esdglider toolbox developed by @smwoodman. My goal is to align this SeaExplorer processing with our existing esdglider workflow to produce consistent data outputs and format the dataset for the glider DAC.

Progress made so far;

• Following the getting started SeaExplorer example I created a new deployment yaml with our sensor specific information. I had to make some changes to account for the legato CTD and PAM sensors but aside from that I think this is pretty similar to the template.
• I also started a processing script based on the example, available here and have the logger output here

Some initial questions;

1. I only have access to the subset glider data (gli.sub) but I do have the raw science data (raw.pld). I see that I can select the kind in the merge_parquet and raw_to_timeseries functions, but can I select different types for glider and science data? Would this make processing too complicated?
2. All of my data files are .gz which I understand is a compressed version. Do I have to decompress them before processing? I tried pointing the code to the .gz files but I got a lot of warnings in the log file that they could not be read (example here). I also tried decompressing one of the files and leaving that in the directory but I'm not sure if that helped.

I'm sure I will have additional questions but hopefully this is a good starting point for discussion.

For additional context, I am relatively new to Python but have more experience working in R so I can hobble along for most scripts but very open to all suggestions for improvement!

[jklymak]

Hi @Kourtney-Burger

2. the .gz files should decompress fine - its just calling polars.read_csv and that handles *.gz files fine.
3. Your warning look like a file encoding problem. Are you running on a Windows machine (I somewhat guess you are from all the back slashes in your file names). I've not tried to run pyglider on Windows, and it's possible we will have to make some changes for this to work for you. Conversely, it's possible your files just have a non default encoding that I wasn't aware of and you came across it. Can you make a few files available?

but can I select different types for glider and science data?

I think there is only .sub data for the .gli files anyway? At least that is how our gliders work. seaexplorer.merge_parquet(rawncdir, rawncdir, deploymentyaml, kind='raw') is what leads the processing to use raw pld files instead of the subset ones.

Hope this helps.

[Kourtney-Burger]

Hi Jody,

I am working on a windows machine. Here are some test files

I didn't recover/offload data so I wasn't sure if I was just missing the raw.gli files, but that makes more sense that they may not exist. I currently have the raw.pld. files in a different directory to avoid any issues so I will move them and try to reprocess with kind='raw'.

Thanks!

[jklymak]

Those test files work fine for me::

import logging
import os
import pyglider.seaexplorer as seaexplorer
import pyglider.ncprocess as ncprocess
import polars as pl
import numpy as np
from pathlib import Path

logging.basicConfig(level='INFO')

rawdir  = './delayed_raw/'
rawncdir     = './delayed_rawnc/'
deploymentyaml = ['./SEA117-M026_20260128.yaml']

seaexplorer.raw_to_rawnc(rawdir, rawncdir, deploymentyaml, incremental=False)

# remove some early bad files:
#os.system('rm ' + rawncdir + '/*0001.*')

# merge individual netcdf files into single netcdf files *.nc
seaexplorer.merge_parquet(rawncdir, rawncdir, deploymentyaml, kind='raw')

l1tsdir = './L0-timeseries/'
fnamesuffix = '_delayed'

outname = seaexplorer.raw_to_timeseries(rawncdir, l1tsdir,
    deploymentyaml, kind='raw', profile_filt_time=30,
    profile_min_time=100, fnamesuffix=fnamesuffix,
    replace_attrs={'data_mode': 'P'})

only has an error on the last step because of lack of lat/lon data in the two files you gave me.

So, this is a little hard for me to debug without a Windows machine.

I've made a patch that may work. You'd either need to edit the library on your machine, or clone pyglider and make an editable install of pyglider. https://pyglider.readthedocs.io/en/latest/Install.html#editable-installation