This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Postprocessing

Working with data recorded using ORCA

ORCA provides pre-processing and post-processing code that supports operation of SDRs as chirped radar systems. These functions are meant to provide basic support and analysis for ice-penetrating radar data. More advanced processing can be built off this code. While we built this code to process data collected with radars running ORCA, we hope that the documentation makes it flexible enough to process other forms of ice-penetrating radar data as well.

Data Format

Pre-processing Functions

Pre-processing functions available in ORCA are centered around waveform (chirp) generation. They are found in the preprocessing/ folder of the main repo.

  • preprocessing/generate_chirp.py Generates digital IQ (in-phase and quadrature) samples based on configuration parameters provided in a YAML configuration file
    • Support for the following window functions is currently provided: Rectangular, Hamming, Blackman-Harris, Kaiser10, Kaiser14, and Kaiser18
    • We have found this document by G. Heinzel and others to be a very useful resource for understanding and implementing window functions

Post-processing Functions

A variety of post-processing functions are available in ORCA, ranging from basic radar processing, to radar system coherence characterization.

  • processing.py
  • processing_dask.py
  • save_data.py
  • merge_data.py

Post-processing Notebooks

We provide several starter notebooks to enable radar data processing and system characterization.

  • notebooks/Field Processing.ipynb A basic one stop radar processing code notebook, intended to produce ample information for in-field (or lab) debugging
    • Loads a binary file containing IQ radar data and resaves it into Zarr format
    • Loads the metadata associated with that binary file (YAML configuration file and log output)
    • Generates a reference chirp (by calling preprocessing/generate_chirp.py) to use for pulse compression
    • Displays a single received pulse in the time domain
    • Removes errors (if necessary) from the collected radar data
    • Coherently averages (stacks) the data based on user-input stacking parameter
    • Pulse compresses the data
    • Displays a 1D radargram of the first and last pulse compressed pulse
    • Displays a 2D radargram of the entire dataset
    • Displays spectrogram (fast-time) of received data
    • Displays power spectrum of received data
  • notebooks/Radar 1D File Compare.ipynb A notebook to compare 1D pulse compressed radar data in different files
    • Facilitates comparison between different radar configurations
  • notebooks/Radar 1D Stacking.ipynb A notebook to display a 1D radargram with different amounts of stacking
    • Facilitates comparison of stacking benefits and a quick check on coherence of data
  • notebooks/Radar Spectrogram.ipynb A notebook to display a fast time and slow time spectrogram of radar data
    • Facilitates debugging of RFI, frequency spurs, and other noise sources
  • notebooks/Dask Noise Statistics.ipynb A notebook to compute and display mean noise power and variance for radar data
    • Facilitates analysis of coherence by computing the mean noise power and variance for different amounts of radar stacking

The notebooks/orca_paper subdirectory contains notebooks used to produce the figures in our IEEE paper:

T. O. Teisberg, A. L. Broome and D. M. Schroeder, “Open Radar Code Architecture (ORCA): A Platform for Software-Defined Coherent Chirped Radar Systems,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1-11, 2024, Art no. 5109411, doi: 10.1109/TGRS.2024.3446368.

1 - File Formats

How data is stored from the ORCA system

2 - Processing Functions

Basic postprocessing functions for working with radar data

3 - Postprocessing Notebooks

An overview of ready-to-go notebooks for processing your data

4 - Timing and timestamping

How recorded samples are timestamped and options for synchronization

Timestamping of received data is relatively basic and not yet suitable for high-precision applications. This page describes the basics as currently implemented.

There is significant room to improve on ORCA here. USRP devices are capable of multiple means of synchronization between devices that could be used to provide better timestamping.

Relative timing of samples

Transmit and receive events are scheduled as timed commands. Generally, bursts of samples are recorded at intervals specified by pulse_rep_int.

When an error occurs, extra time is added to allow the system to catch up. The amount of time added is configured by the error_recovery_time parameter. When time is added, a message is printed to stdout with the chirp number and the amount of time added, such as the following:

[1691434273.020] 	[TX] (Chirp 975541) time_offset increased by 0.0016

Note that due to the enqueueing of commands, multiple errors may occur before the next set of commands is enqueued with a time offset. If more than one error occurs, the time will be offset by number of errors * error_recovery_time.

These output messages are read during the Zarr file creation process in order to provide a corrected slow time record.

Timestamping samples

ORCA currently assumes that the SDR clock is useful to relative timing only and relies on the host computer clock to provide a starting timestamp. When using the run.py interface, stdout output is logged with a Unix timestamp from the host computer. The timestamp on the starting message (plus any specified time_offset in the config file) is used to provide the absolute time of the first transmission.

This relies upon the system clock being set accurately and should never be assumed to provide sub-second accuracy.

Accessing timestamps in Xarray

The default coordiantes of the radar DataArray are sample_idx and pulse_idx. These correspond to the fast time and slow time indices of the as-recorded data.

Considering pulse_idx this means, for example, that if num_presums is 10, then there are 10 chirps transmitted and received for each index of pulse_idx. Similarly, any errors are ignored in counting pulse_idx.

The additional data variables chirp_sequence_idx, slow_time, and timestamp may be used if other information is needed. They are as follows:

  • chirp_sequence_idx refers to the index as tracked by the SDR of the first chirp included in the associated data recording. With no errors, chirp_sequence_idx = num_presums * pulse_idx.
  • slow_time refers to the relative time (in seconds) of the start of the first chirp included in the assocaited data recording.
  • timestamp is an np.datetime64 series equal to slow_time + start timestamp