sage.data.primer.get_data_release

Filename : real_noise.py Description : Short description of the file

Created on 2025-11-06 15:00:16

__author__ = Narenraju Nagarajan __copyright__ = Copyright 2025, ProjectName __license__ = MIT Licence __version__ = 0.0.1 __maintainer__ = Narenraju Nagarajan __affiliation__ = N/A __email__ = N/A __status__ = [‘inProgress’, ‘Archived’, ‘inUsage’, ‘Debugging’]

GitHub Repository: NULL

Documentation: NULL

Classes

DownloadConfig	Immutable, multiprocessing-safe download configuration.
DataReleaseDownloader	Parallel GWOSC data downloader that stores validated strain segments to HDF5.

Functions

validate_segment(bin_path, seg_meta, *[, strict])	Verify the checksum of a single binary segment.
validate_all_segments(bin_path, metadata)	Verify checksums for all segments in metadata.
file_checksum(path[, algo, block])	Compute the checksum of an entire file using streaming reads.

Module Contents

class DownloadConfig

Immutable, multiprocessing-safe download configuration.

Frozen dataclass passed to worker processes via multiprocessing Pool.starmap; must be pickleable (no open file handles, no locks).

sample_rate

Target sample rate (Hz) after downsampling.

Type:

float

trim

Seconds of corrupt data to trim from each edge after download.

Type:

float

noise_low_freq_cutoff

Low-frequency cutoff (Hz) used during downsampling.

Type:

float

max_retries

Maximum number of download attempts before giving up on a segment.

Type:

int

delay

Seconds to wait between retry attempts.

Type:

float

sample_rate: float

trim: float

noise_low_freq_cutoff: float

max_retries: int

delay: float

validate_segment(bin_path, seg_meta, *, strict=True)

Verify the checksum of a single binary segment.

Parameters:

• bin_path (str or Path) – Path to the .bin file.

• seg_meta (dict) – Segment metadata dict from the sidecar JSON (must contain "checksum_algo", "byte_offset", "byte_length", "checksum", and "segment_index").

• strict (bool) – If True (default), raise IOError on mismatch.

Returns:

True if the checksum matches, False otherwise.

Return type:

bool

validate_all_segments(bin_path, metadata)

Verify checksums for all segments in metadata.

Parameters:

• bin_path (str or Path) – Path to the .bin file.

• metadata (list[dict]) – List of segment metadata dicts as produced by the sidecar JSON.

Returns:

Segment indices that failed the checksum. An empty list means all segments are intact.

Return type:

list[int]

file_checksum(path, algo='sha256', block=1 << 20)

Compute the checksum of an entire file using streaming reads.

Parameters:

• path (str or Path) – File to hash.

• algo (str) – Hash algorithm name (default "sha256").

• block (int) – Read block size in bytes (default 1 MiB).

Returns:

Hex-encoded digest string.

Return type:

str

class DataReleaseDownloader(segments_metadata, save_parent_dir, noise_low_freq_cutoff=15.0, minimum_segment_duration=22.0, corrupt_trim_length=0.2, max_download_retries=10, retry_delay=0.5, num_workers=4, proxy_reset_every=50, proxy_reset_sleep=90.0, make_monolithic_file=True, save_bin=False, sample_rate=2048.0, release_dirname='data_release')

Parallel GWOSC data downloader that stores validated strain segments to HDF5.

Downloads LIGO/Virgo/KAGRA open-data strain segments from GWOSC for a structured-array of (detector, observing-run, segment-list) records. Supports both a monolithic HDF5 file and a raw binary output format, with optional checksum validation per segment.

Parameters:

• segments_metadata (np.ndarray (SEGMENT_DTYPE)) – Structured array produced by TimelineQuery.

• save_parent_dir (str) – Root directory under which the data_release/ sub-folder is created.

• noise_low_freq_cutoff (float) – High-pass cutoff applied during downsampling (default 15.0 Hz).

• minimum_segment_duration (float) – Segments shorter than this (seconds, post-trim) are discarded (default 22.0).

• corrupt_trim_length (float) – Seconds trimmed from each edge to discard corrupted data (default 0.2).

• max_download_retries (int) – Maximum number of GWOSC fetch retries per segment (default 10).

• retry_delay (float) – Seconds between retries (default 0.5).

• num_workers (int) – Number of parallel worker processes (default 4).

• make_monolithic_file (bool) – If True (default), all segments for a (detector, run) pair are concatenated into a single HDF5; otherwise one file per chunk.

• save_bin (bool) – If True, write raw binary (float32 LE) instead of HDF5, with a companion JSON segment index (default False).

• sample_rate (float) – Target sample rate after downsampling (default 2048.0 Hz).

• proxy_reset_every (int)

• proxy_reset_sleep (float)

• release_dirname (str)

sample_rate = 2048.0

noise_low_freq_cutoff = 15.0

trim = 0.2

minimum_segment_duration = 22.0

max_retries = 10

delay = 0.5

num_workers = 4

proxy_reset_every = 50

proxy_reset_sleep = 90.0

save_parent_dir

release_dirname = 'data_release'

monolithic = True

full_metadata

dcfg

save_bin = False

retry_and_append(bin_path, seg_json_path, segments_to_retry, detector, run)

Download missing/failed segments and append them to an existing .bin file.

Parameters:

• bin_path (Path or str) – Path to the existing monolithic .bin file.

• seg_json_path (Path or str) – Path to the existing *_segments.json sidecar.

• segments_to_retry (array-like of shape (N, 2)) – Pre-trim GPS start/end times for each segment to retry.

• detector (str) – Detector name (e.g. "H1").

• run (str) – Observing run label (e.g. "O3b").

• appended (After this call seg_json_path is updated with new entries)

• still (and a *_retry_failed.json is written with any segments that)

• downloaded. (could not be)

download()

Download all segments from GWOSC and save to disk.

Iterates over every (detector, observing-run, segments) record in full_metadata, calling _fetcher() for each. Records with None value are skipped silently. Output is written to the data_release/ sub-directory under save_parent_dir.