airbornegeo.filter_line

Contents

airbornegeo.filter_line#

filter_line(data, *, filter_width, data_column, filter_by_column, filter_type='lowpass', filter_shape='gaussian', engine='scipy', groupby_column=None, progressbar=True, pad_width_percentage=10, pad_mode='reflect', max_gap=None, robust=False, robust_threshold=2.5, **kwargs)[source]#

Apply a 1D filter to a column of a pandas DataFrame along values of another column. The filter_by_column would typically be either distance along track for a spatial filter, or a time column, for a temporal filter. The dataframe can be grouped by the groupby_column before applying the filter. This column could contain flight names, or lines names. filter_width is the filter width in the same units as filter_by_column (e.g. if filter_by_column is “distance_along_line” in meters, a filter_width of 1000 gives a 1000 m filter; if filter_by_column is a time column in seconds, filter_width is in seconds). Ends of lines are automatically padded to avoid edge effects.

Two engines are available, selected with the engine parameter:

  • “scipy” (the default): applies the filter directly in Python via scipy, with no external dependencies. It additionally splits each line at gaps in filter_by_column larger than max_gap and filters the segments separately, so anomalies are not smeared across genuine breaks in the data (see max_gap). This engine is usually substantially faster than “gmt”, especially when filtering many lines/segments individually with groupby_column, since GMT’s filter1d has a large fixed per-call overhead.

  • “gmt”: applies the filter with GMT’s filter1d, via the optional pygmt package (raises an ImportError with installation instructions if pygmt is not installed). This is a mature, independently-validated reference implementation; use it if you need results that exactly reproduce GMT’s own processing.

Both engines aim to give equivalent (though not always numerically identical) results for the same filter_shape/filter_width/filter_type/robust combination.

Parameters:

  • data (DataFrame) – Dataframe containing the data points to filter.

  • filter_width (float) – The filter width, in the same units as filter_by_column. For “gaussian”, this is the full width (6 standard deviations); for the others, the full window width.

  • data_column (str) – The data to filter.

  • filter_by_column (str) – The independent variable to filter against, typically either a time or distance along track values.

  • filter_type (str) – Either “lowpass” or “highpass”, by default “lowpass”.

  • filter_shape (str) – One of “gaussian”, “boxcar”, “cosine”, or “median”, by default “gaussian”. See the “scipy” engine’s implementation for what each of these represents; “gmt” reproduces the equivalent pygmt.filter1d filter type (“g”, “b”, “c”, “m” respectively).

  • engine (str) – Either “scipy” or “gmt”, by default “scipy”.

  • groupby_column (str | None) – Column name to group by before filtering, by default None.

  • progressbar (bool) – Show progress bar for each group, by default True

  • pad_width_percentage (float) – The width of the pad to add before and after the data in percentage of the range of values provided by filter_by_column, by default 10.

  • pad_mode (str) – The mode to use for padding, by default is “reflect”.

  • max_gap (float | None) – Only used by engine “scipy”: split lines where consecutive filter_by_column values differ by more than this (in the units of filter_by_column) and filter each segment separately. By default (None) gaps larger than 10 times the median spacing are split; use numpy.inf to disable splitting.

  • robust (bool) – Use a robust variant of the filter that resists outliers, by default False. For engine “scipy”, this despikes the data before filtering (replacing samples far from the local median with that median) and is not supported for filter_shape “median”, which is already robust by construction. For engine “gmt”, this uses GMT’s own uppercase (robust) filter codes.

  • robust_threshold (float) – Only used by engine “scipy” when robust is True: samples deviating from the local median by more than robust_threshold times the local (MAD-based) robust standard deviation are replaced by that median before filtering, by default 2.5 (matching GMT’s default for its own robust filters).

  • kwargs (Any) – Keyword arguments to pass to np.pad, such as stat_length and constant_values.

Returns:

The filtered data values

Return type:

Series

Contents