Fileio

source module isofit.core.fileio

Classes

• SpectrumFile — A buffered file object that contains configuration information about formatting, etc.

• InputData

• IO — ...

Functions

• write_bil_chunk — Write a chunk of data to a binary, BIL formatted data cube. Args: dat: data to write outfile: output file to write to line: line of the output file to write to shape: shape of the output file dtype: output data type

• initialize_output — Initialize output file by updating metadata and creating object.

source class SpectrumFile(fname, write=False, n_rows=None, n_cols=None, n_bands=None, interleave=None, dtype=np.float32, wavelengths=None, fwhm=None, band_names=None, bad_bands='[]', zrange='{0.0, 1.0}', flag=-9999.0, ztitles='{Wavelength (nm), Magnitude}', map_info='{}', engine_name=None, isofit_version=None)

A buffered file object that contains configuration information about formatting, etc.

.

Methods

• open_map_with_retries — Try to open a memory map, handling Beowulf I/O issues.

• get_frame — The frame is a 2D array, essentially a list of spectra. The self.frames list acts as a hash table to avoid storing the entire cube in memory. So we read them or create them on an as-needed basis. When the buffer flushes via a call to flush_buffers, they will be deleted.

• write_spectrum — We write a spectrum. If a binary format file, we simply change the data cached in self.frames and defer file I/O until flush_buffers is called.

• read_spectrum — Get a spectrum from the frame list or ASCII file. Note that if we are an ASCII file, we have already read the single spectrum and return it as-is (ignoring the row/column indices).

• flush_buffers — Write to file, and refresh the memory map object.

source method SpectrumFile.open_map_with_retries()

Try to open a memory map, handling Beowulf I/O issues.

Raises

• IOError

source method SpectrumFile.get_frame(row)

The frame is a 2D array, essentially a list of spectra. The self.frames list acts as a hash table to avoid storing the entire cube in memory. So we read them or create them on an as-needed basis. When the buffer flushes via a call to flush_buffers, they will be deleted.

source method SpectrumFile.write_spectrum(row, col, x)

We write a spectrum. If a binary format file, we simply change the data cached in self.frames and defer file I/O until flush_buffers is called.

source method SpectrumFile.read_spectrum(row, col)

Get a spectrum from the frame list or ASCII file. Note that if we are an ASCII file, we have already read the single spectrum and return it as-is (ignoring the row/column indices).

source method SpectrumFile.flush_buffers()

Write to file, and refresh the memory map object.

source class InputData()

Methods

• clear

source method InputData.clear()

source class IO(config: Config, forward: ForwardModel, full_statevec: list = [])

...

Initialization specifies retrieval subwindows for calculating measurement cost distributions.

Methods

• get_components_at_index — Load data from input files at the specified (row, col) index.

• flush_buffers — Write all buffered output data to disk, and erase read buffers.

• write_datasets — Write all valid datasets to disk (possibly buffered).

• build_output — Build the output to be written to disk as a dictionary

• write_spectrum — Convenience function to build and write output in one step

• initialize_output_files

source method IO.get_components_at_index(row: int, col: int) → InputData

Load data from input files at the specified (row, col) index.

Parameters

• row : int — row to retrieve data from

• col : int — column to retrieve data from

Returns

• InputData — object containing all current data reads

source method IO.flush_buffers()

Write all buffered output data to disk, and erase read buffers.

source method IO.write_datasets(row: int, col: int, output: dict, states: List, flush_immediately=False)

Write all valid datasets to disk (possibly buffered).

Parameters

• row : int — row to write to

• col : int — column to write to

• output : dict — dictionary with keys corresponding to config.input file references

• states : List — results states from inversion. In the MCMC case, these are interpreted as samples from the

• posterior, otherwise they are a gradient descent trajectory (with the last spectrum being the converged

• solution).

• flush_immediately — IO argument telling us to immediately write to disk, ignoring config settings

source method IO.build_output(states: List, input_data: InputData, fm: ForwardModel, iv: Inversion, fill_value=-9999.0)

Build the output to be written to disk as a dictionary

Parameters

• states : List — results states from inversion. In the MCMC case, these are interpreted as samples from the

• posterior, otherwise they are a gradient descent trajectory (with the last spectrum being the converged

• solution).

• input_data : InputData — an InputData object

• fm : ForwardModel — the forward model used to solve the inversion

• iv : Inversion — the inversion object

Raises

• IOError

source method IO.write_spectrum(row: int, col: int, states: List, fm: ForwardModel, iv: Inversion, flush_immediately=False, input_data: InputData = None)

Convenience function to build and write output in one step

Parameters

• row : int — data row to write

• col : int — data column to write

• states : List — results states from inversion. In the MCMC case, these are interpreted as samples from the

• posterior, otherwise they are a gradient descent trajectory (with the last spectrum being the converged

• solution).

• meas — measurement radiance

• geom — geometry object of the observation

• fm : ForwardModel — the forward model used to solve the inversion

• iv : Inversion — the inversion object

• flush_immediately — IO argument telling us to immediately write to disk, ignoring config settings

• input_data : InputData — optionally overwride self.current_input_data

source staticmethod IO.initialize_output_files(config, n_rows, n_cols, full_statevector)

source write_bil_chunk(dat: np.array, outfile: str, line: int, shape: tuple, dtype: str = 'float32') → None

Write a chunk of data to a binary, BIL formatted data cube. Args: dat: data to write outfile: output file to write to line: line of the output file to write to shape: shape of the output file dtype: output data type

Returns

• None — None

source initialize_output(output_metadata, outpath, out_shape, **kwargs)

Initialize output file by updating metadata and creating object.

Parameters

• output_metadata — dict - Dictionary with envi header information

• outpath — str - path to output file

• out_shape — tuple - dimensions of initialized file

• keys_to_del — list - keys to remove from output_metadata

• kwargs - key-argument pairs to add to output_metadata