Apply OE¶

Apply OE is the built-in end-to-end atmospheric correction pipeline, and is the easiest way to run Isofit. Apply OE is run via a command line interface (CLI) tool accessed with a terminal call:

isofit apply_oe --help

The function docstring is printed when this command is run within a terminal window, and acts as a guide for how to run the function.

In [1]:

from isofit.utils.apply_oe import apply_oe

print(apply_oe.__doc__)

from isofit.utils.apply_oe import apply_oe print(apply_oe.__doc__)

    Applies OE over a flightline using a radiative transfer engine. This executes
    ISOFIT in a generalized way, accounting for the types of variation that might be
    considered typical.

    Observation (obs) and location (loc) files are used to determine appropriate
    geometry lookup tables and provide a heuristic means of determining atmospheric
    water ranges.

    Parameters
    ----------
    input_radiance : str
        Radiance data cube. Expected to be ENVI format
    input_loc : str
        Location data cube of shape (Lon, Lat, Elevation). Expected to be ENVI format
    input_obs : str
        Observation data cube of shape:
            (path length, to-sensor azimuth, to-sensor zenith,
            to-sun azimuth, to-sun zenith, phase,
            slope, aspect, cosine i, UTC time)
        Expected to be ENVI format
    working_directory : str
        Directory to stage multiple outputs, will contain subdirectories
    sensor : str
        The sensor used for acquisition, will be used to set noise and datetime
        settings
    surface_path : str
        Path to surface model or json dict of surface model configuration
    copy_input_files : bool, default=False
        Flag to choose to copy input_radiance, input_loc, and input_obs locally into
        the working_directory
    modtran_path : str, default=None
        Location of MODTRAN utility. Alternately set with `MODTRAN_DIR` environment
        variable
    wavelength_path : str, default=None
        Location to get wavelength information from, if not specified the radiance
        header will be used
    surface_category : str, default="multicomponent_surface"
        The type of ISOFIT surface priors to use.  Default is multicomponent_surface
    aerosol_climatology_path : str, default=None
        Specific aerosol climatology information to use in MODTRAN
    rdn_factors_path : str, default=None
        Specify a radiometric correction factor, if desired
    atmosphere_type : str, default="ATM_MIDLAT_SUMMER"
        Atmospheric profile to be used for MODTRAN and libRadtran simulations only.
        However, if presolve mode enabled this is used to inform max water
        column vapor for any radiative transfer model. Valid options include:
        ATM_MIDLAT_SUMMER, ATM_TROPICAL, ATM_MIDLAT_WINTER,
        ATM_SUBARC_SUMMER, ATM_SUBARC_WINTER, or ATM_US_STANDARD_1976.
    channelized_uncertainty_path : str, default=None
        Path to a wavelength-specific channelized uncertainty file - used to augment Sy in the OE formalism
    instrument_noise_path : str, default=None
        Path to a wavelength-specific instrument noise file, used to derive per-wavelength NEDL / SNR
    dn_uncertainty_file:  str, default=None
        Path to a linearity .mat file to augment S matrix with linearity uncertainty
    model_discrepancy_path : str, default=None
        Modifies S_eps in the OE formalism as the Gamma additive term, as:
        S_eps = Sy + Kb.dot(self.Sb).dot(Kb.T) + Gamma
    lut_config_file : str, default=None
        Path to a look up table configuration file, which will override defaults
        choices
    multiple_restarts : bool, default=False
        Use multiple initial starting poitns for each OE ptimization run, using
        the corners of the atmospheric variables as starting points.  This gives
        a more robust, albeit more expensive, solution.
    logging_level : str, default="INFO"
        Logging level with which to run ISOFIT
    log_file : str, default=None
        File path to write ISOFIT logs to
    n_cores : int, default=1
        Number of cores to run ISOFIT with. Substantial parallelism is available, and
        full runs will be very slow in serial. Suggested to max this out on the
        available system
    presolve : int, default=False
        Flag to use a presolve mode to estimate the available atmospheric water range.
        Runs a preliminary inversion over the image with a 1-D LUT of water vapor, and
        uses the resulting range (slightly expanded) to bound determine the full LUT.
        Advisable to only use with small cubes or in concert with the empirical_line
        setting, or a significant speed penalty will be incurred
    empirical_line : bool, default=False
        Use an empirical line interpolation to run full inversions over only a subset
        of pixels, determined using a SLIC superpixel segmentation, and use a KDTREE of
        local solutions to interpolate radiance->reflectance. Generally a good option
        if not trying to analyze the atmospheric state at fine scale resolution.
        Mutually exclusive with analytical_line
    analytical_line : bool, default=False
        Use an analytical solution to the fixed atmospheric state to solve for each
        pixel.  Starts by running a full OE retrieval on each SLIC superpixel, then
        interpolates the atmospheric state to each pixel, and closes with the
        analytical solution.
        Mutually exclusive with empirical_line
    ray_temp_dir : str, default="/tmp/ray"
        Location of temporary directory for ray parallelization engine
    emulator_base : str, default=None
        Location of emulator base path. Point this at the 3C or 6C .h5 files.
        sRTMnet to use the emulator instead of MODTRAN. An additional file with the
        same basename and the extention _aux.npz must accompany
        e.g. /path/to/emulator.h5 /path/to/emulator_aux.npz
    segmentation_size : int, default=40
        If empirical_line is enabled, sets the size of segments to construct
    num_neighbors : list[int], default=[]
        Forced number of neighbors for empirical line extrapolation - overides default
        set from segmentation_size parameter
    atm_sigma : list[int], default=[2]
        A list of smoothing factors to use during the atmospheric interpolation, one
        for each atmospheric parameter (or broadcast to all if only one is provided).
        Only used with the analytical line.
    pressure_elevation : bool, default=False
        Flag to retrieve elevation
    prebuilt_lut : str, default=None
        Use this pre-constructed look up table for all retrievals. Must be an
        ISOFIT-compatible RTE NetCDF
    no_min_lut_spacing : bool, default=False
        Don't allow the LUTConfig to remove a LUT dimension because of minimal spacing.
    inversion_windows : list[float], default=None
        Override the default inversion windows.  Will supercede any sensor specific
        defaults that are in place.
        Must be in 2-item tuples
    config_only : bool, default=False
        Generates the configuration then exits before execution. If presolve is
        enabled, that run will still occur.
    interpolate_bad_rdn : bool, default=False
        Flag to perform a per-pixel interpolation across no-data and NaN data bands.
        Does not interpolate vectors that are entire no-data or NaN, only partial.
        Currently only designed for wavelength interpolation on spectra.
        Does NOT do any spatial interpolation
    interpolate_inplace : bool, default=False
        Flag to tell interpolation to work on the file in place, or generate a
        new interpolated rdn file. The location of the new file will be in the
        "input" directory within the working directory.
    skyview_factor : str, default=None
        Flag to determine method to account for skyview factor. Default is None, creating an array of 1s.
        Other option is "slope" which will approx. based on cos^2(slope/2).
        Other option is a path to a skyview ENVI file computed via skyview.py utility or other source.
        Please note data must range from 0-1.
    resources : bool, default=False
        Enables the system resource tracker. Must also have the log_file set.
    retrieve_co2 : bool, default=False
        Flag to retrieve CO2 in the state vector. Only available with emulator at the moment.
    eof_path : str, default=None
        Add 1 or 2 Empirical Orthogonal Functions to the state vector.  File is a 1-2 column text file
        with one number per instrument channel.
    terrain_style : str, default=dem
        Flag to set the terrain style.  dem uses provided obs values, flat sets the surface to the spheroid

    References
    ----------
    D.R. Thompson, A. Braverman,P.G. Brodrick, A. Candela, N. Carbon, R.N. Clark,D. Connelly, R.O. Green, R.F.
    Kokaly, L. Li, N. Mahowald, R.L. Miller, G.S. Okin, T.H.Painter, G.A. Swayze, M. Turmon, J. Susilouto, and
    D.S. Wettergreen. Quantifying Uncertainty for Remote Spectroscopy of Surface Composition. Remote Sensing of
    Environment, 2020. doi: https://doi.org/10.1016/j.rse.2020.111898.

    sRTMnet emulator:
    P.G. Brodrick, D.R. Thompson, J.E. Fahlen, M.L. Eastwood, C.M. Sarture, S.R. Lundeen, W. Olson-Duvall,
    N. Carmon, and R.O. Green. Generalized radiative transfer emulation for imaging spectroscopy reflectance
    retrievals. Remote Sensing of Environment, 261:112476, 2021.doi: 10.1016/j.rse.2021.112476.
    

The Apply OE function can leverage a large number of input parameters, but most are optional. The important inputs are the non-optional arguments:

INPUT_RADIANCE INPUT_LOC INPUT_OBS WORKING_DIRECTORY SENSOR and --surface_path.

which must be entered in the specified order (except --surface_path which can be added at any point but still required). Descriptions of each are found in the docstring printed above. It is important to note that the INPUT_RADIANCE, INPUT_LOC, and INPUT_OBS are ENVI raster data formats that must be at the same row-column dimensions. The --surface_path points Isofit towards the constructed prior distribution file for surface reflectance. Optional arguments are denoted by the '--' in their name, e.g. --modtran_path, --pressure_elevation. It is important to note that the default radiative transfer engine (RTE) is currently set to sRTMnet, set via --emulator_path.

INPUT_RADIANCE¶

The data is at-sensor radiance in units of $\mu W \, cm^{-2} \, nm^{-1} \, sr^{-1} $. You may leverage isofit.core.units module which offers many common conversions. The .hdr file should be used to store the wavelength and fwhm.

INPUT_LOC¶

1. Longitude [degrees]

   • Range is -180 to 180

2. Latitude [degrees]

   • Range is -90 to 90

3. Elevation [meters]

   • Above mean sea level

INPUT_OBS¶

Note: the names of the keys do not need to match exactly, only the order matters.

1. Path length [meters]

   • Defined as the slant path from the sensor to the surface

2. To-sensor azimuth, $\phi_v$ [degrees]

   • Ranges from 0-360, measured clockwise from north

3. To-sensor zenith, $\theta_v$ [degrees]

   • Ranges from 0-90, measured from zenith

4. To-sun azimuth, $\phi_s$ [degrees]

   • Ranges from 0-360, measured clockwise from north

5. To-sun zenith, $\theta_s$ [degrees]

   • Ranges from 0-90, measured from zenith

6. Phase angle, $\xi$ [degrees]

   • $cos(\xi) = cos(\theta_s) \, cos(\theta_v) \, + \, sin(\theta_s) \, sin(\theta_v) \, cos(\phi_s - \phi_v)$
   • $\xi = arccos(cos(\xi))$

7. Slope, $S$ [degrees]

   • Ranges from 0-90

8. Aspect, $A$ [degrees]

   • Ranges from 0-360, measured clockwise from north

9. Cosine of the local solar illumination angle, $cos(i)$ [unitless]

   • $cos(i) = cos(\theta_s) \, cos(S) \, + \, sin(\theta_s) \, sin(S) \, cos(\phi_s - A) $

10. UTC time [decimal]

    • Ranges from 0 - 24

11. Earth-sun distance [AU]

    • Astronomical Unit as a function of day of year

How do you run Apply OE?¶

The script is run via the CLI. For example:

IC=$(isofit path examples)/image_cube/small sets the environment variable IC to the output of the command isofit path examples/image_cube/small. This is just used to reuse the same output for multiple arguments. Depending on the user's isofit installation environment, this path may vary. This command ensures the correct path is retrieved

  IC=$(isofit path examples)/image_cube/small EX=$(isofit path examples)/image_cube/small \
  isofit apply_oe \
  $IC/ang20170323t202244_rdn_7000-7010 \
  $IC/ang20170323t202244_loc_7000-7010 \
  $IC/ang20170323t202244_obs_7000-7010 \
  $EX \
  ang \
  --surface_path $EX/configs/surface.json \
  --emulator_base $(isofit path srtmnet --key file) \
  --n_cores 10 \
  --presolve \

Here,

$IC/ang20170323t202244_rdn_7000-7010
$IC/ang20170323t202244_loc_7000-7010
$IC/ang20170323t202244_obs_7000-7010

are the radiance, location, and observational geometry files respectively. The \ tells the CLI call to expect a multi-line input. The remaining two required parameters are ang, the sensor designation (AVIRIS-NG), and the --surface_path pointing to the surface configuration file which will be built at runtime.

The remaining arguments set Apply OE to run with:

1. --emulator_base points isofit to the location of the sRTMnet emulator to usse as the radiative transfer engine (RTE)
2. --n_cores = 10 CPU cores
3. The --presolve algorithm to narrow down the water vapor retrievals

In [2]:

# Alternatively, you can programatically call apply_oe. If you executed apply_oe via the command line above, do not run this
# We use the env object to retrieve the actual location of the ISOFIT extra dependencies instead of assuming where they could be
import os
import shutil

from isofit.utils.apply_oe import apply_oe
from isofit.data import env

output = env.path("examples", "image_cube", "small")
logfile = env.path("examples", "image_cube", "small") / "log.txt"

# Cleanup any previous runs; comment this out if you want to preserve a previous run's output
if (o := output / "output").exists():
    shutil.rmtree(o)

apply_oe(
    input_radiance    = str(env.path("examples", "image_cube", "small", "data", "ang20170323t202244_rdn_7000-7010")),
    input_loc         = str(env.path("examples", "image_cube", "small", "data", "ang20170323t202244_loc_7000-7010")),
    input_obs         = str(env.path("examples", "image_cube", "small", "data", "ang20170323t202244_obs_7000-7010")),
    working_directory = str(output),
    sensor            = "ang",
    surface_path      = str(env.path("examples", "image_cube", "small", "configs", "surface.json")),
    emulator_base     = str(env.path("srtmnet", key="srtmnet.file")),
    presolve          = True,
    n_cores           = os.cpu_count(),
)

# Alternatively, you can programatically call apply_oe. If you executed apply_oe via the command line above, do not run this # We use the env object to retrieve the actual location of the ISOFIT extra dependencies instead of assuming where they could be import os import shutil from isofit.utils.apply_oe import apply_oe from isofit.data import env output = env.path("examples", "image_cube", "small") logfile = env.path("examples", "image_cube", "small") / "log.txt" # Cleanup any previous runs; comment this out if you want to preserve a previous run's output if (o := output / "output").exists(): shutil.rmtree(o) apply_oe( input_radiance = str(env.path("examples", "image_cube", "small", "data", "ang20170323t202244_rdn_7000-7010")), input_loc = str(env.path("examples", "image_cube", "small", "data", "ang20170323t202244_loc_7000-7010")), input_obs = str(env.path("examples", "image_cube", "small", "data", "ang20170323t202244_obs_7000-7010")), working_directory = str(output), sensor = "ang", surface_path = str(env.path("examples", "image_cube", "small", "configs", "surface.json")), emulator_base = str(env.path("srtmnet", key="srtmnet.file")), presolve = True, n_cores = os.cpu_count(), )

INFO:2026-05-20,21:20:58 || apply_oe.py:apply_oe() | Checking input data files...

INFO:2026-05-20,21:20:58 || apply_oe.py:apply_oe() | ...Data file checks complete

INFO:2026-05-20,21:20:58 || apply_oe.py:apply_oe() | Setting up files and directories....

INFO:2026-05-20,21:20:58 || template_construction.py:__init__() | Flightline ID: ang20170323t202244

INFO:2026-05-20,21:20:58 || template_construction.py:__init__() | no noise path found, proceeding without

INFO:2026-05-20,21:20:58 || apply_oe.py:apply_oe() | ...file/directory setup complete

INFO:2026-05-20,21:20:58 || apply_oe.py:apply_oe() | Using inversion windows: [[350.0, 1360.0], [1410, 1800.0], [1970.0, 2500.0]]

INFO:2026-05-20,21:20:58 || template_construction.py:get_wavelengths() | No wavelength file provided. Obtaining wavelength grid from ENVI header of radiance cube.

INFO:2026-05-20,21:20:58 || template_construction.py:get_wavelengths() | Wavelength units of nm inferred...converting to microns

INFO:2026-05-20,21:20:58 || template_construction.py:check_surface_model() | No surface model provided. Build new one using given config file.

0 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

1 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

2 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']
3 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

4 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

5 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

6 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']
7 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | Observation means:

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | Path (km): 20.716992087408016

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | To-sensor azimuth (deg): 71.27130737250893

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | To-sensor zenith (deg): 17.34528282852098

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | To-sun azimuth (deg): 197.3237574125862

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | To-sun zenith (deg): 32.45698605676816

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | Relative to-sun azimuth (deg): 126.05245004007726

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | Altitude (km): 19.808562728369008

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | Pre-solve H2O grid: [0.2  0.76 1.33 1.89 2.46 3.02 3.59 4.15 4.71 5.28]

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | Writing H2O pre-solve configuration file.

INFO:2026-05-20,21:21:01 || ini.py:toTemplate() | Saved converted json to: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/config/ang20170323t202244_h2o.json.tmpl

INFO:2026-05-20,21:21:01 || apply_oe.py:apply_oe() | Run ISOFIT initial guess

WARNING:2026-05-20,21:21:01 || __init__.py:checkNumThreads() | 
******************************************************************************************
! Number of threads is greater than 1 (currently: 4), this may greatly impact performance
! Please set this the environment variables 'MKL_NUM_THREADS' and 'OMP_NUM_THREADS' to '1'
******************************************************************************************

INFO:2026-05-20,21:21:01 || configs.py:create_new_config() | Loading config file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/config/ang20170323t202244_h2o.json

INFO:2026-05-20,21:21:01 || configs.py:get_config_errors() | Checking config sections for configuration issues

INFO:2026-05-20,21:21:01 || configs.py:get_config_errors() | Configuration file checks complete, no errors found.

/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)

2026-05-20 21:21:07,200	INFO worker.py:2012 -- Started a local Ray instance.

/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
  warnings.warn(

INFO:2026-05-20,21:21:09 || isofit.py:run() | Running surfaces: uniform_surface

/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
INFO:2026-05-20,21:21:09 || radiative_transfer_engine.py:__init__() | Loading from wavelength_file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/data/wavelengths.txt

INFO:2026-05-20,21:21:09 || radiative_transfer_engine.py:__init__() | No LUT store found, beginning initialization and simulations

INFO:2026-05-20,21:21:09 || radiative_transfer_engine.py:__init__() | Initializing LUT file

INFO:2026-05-20,21:21:10 || radiative_transfer_engine.py:runSimulations() | Running any pre-sim functions

INFO:2026-05-20,21:21:10 || sRTMnet.py:preSim() | Creating a simulator configuration

INFO:2026-05-20,21:21:10 || sRTMnet.py:preSim() | Building simulator and executing (6S)

INFO:2026-05-20,21:21:10 || radiative_transfer_engine.py:__init__() | No LUT store found, beginning initialization and simulations

INFO:2026-05-20,21:21:10 || radiative_transfer_engine.py:__init__() | Initializing LUT file

INFO:2026-05-20,21:21:10 || radiative_transfer_engine.py:runSimulations() | Running any pre-sim functions

INFO:2026-05-20,21:21:10 || radiative_transfer_engine.py:runSimulations() | Executing parallel simulations

INFO:2026-05-20,21:21:18 || common.py:__call__() |  20.00% simulations complete (elapsed: 0:00:08.043788, rate: 0:00:00.804379, eta: 0:01:12.394092)

INFO:2026-05-20,21:21:18 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:18 || common.py:__call__() |  30.00% simulations complete (elapsed: 0:00:08.285643, rate: 0:00:00.828564, eta: 0:00:33.142572)

INFO:2026-05-20,21:21:18 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:18 || common.py:__call__() |  40.00% simulations complete (elapsed: 0:00:08.524663, rate: 0:00:00.852466, eta: 0:00:19.890880)

INFO:2026-05-20,21:21:18 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:21 || common.py:__call__() |  50.00% simulations complete (elapsed: 0:00:11.701876, rate: 0:00:01.170188, eta: 0:00:17.552814)

INFO:2026-05-20,21:21:21 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:22 || common.py:__call__() |  60.00% simulations complete (elapsed: 0:00:12.200159, rate: 0:00:01.220016, eta: 0:00:12.200159)

INFO:2026-05-20,21:21:22 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:22 || common.py:__call__() |  70.00% simulations complete (elapsed: 0:00:12.413698, rate: 0:00:01.241370, eta: 0:00:08.275799)

INFO:2026-05-20,21:21:22 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:22 || common.py:__call__() |  80.00% simulations complete (elapsed: 0:00:12.591059, rate: 0:00:01.259106, eta: 0:00:05.396168)

INFO:2026-05-20,21:21:22 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:24 || common.py:__call__() |  90.00% simulations complete (elapsed: 0:00:14.362269, rate: 0:00:01.436227, eta: 0:00:03.590567)

INFO:2026-05-20,21:21:24 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:24 || common.py:__call__() | 100.00% simulations complete (elapsed: 0:00:14.623382, rate: 0:00:01.462338, eta: 0:00:01.624820)

INFO:2026-05-20,21:21:24 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:21:25 || radiative_transfer_engine.py:runSimulations() | Running any post-sim functions

INFO:2026-05-20,21:21:25 || radiative_transfer_engine.py:runSimulations() | Saving post-sim data to index zero of all dimensions except wl

INFO:2026-05-20,21:21:26 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:21:26 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:21:26 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:21:26 || sRTMnet.py:preSim() | Interpolating simulator quantities to emulator size

INFO:2026-05-20,21:21:26 || sRTMnet.py:preSim() | Loading and predicting with emulator on 4 cores

INFO:2026-05-20,21:21:27 || radiative_transfer_engine.py:runSimulations() | Saving pre-sim data to index zero of all dimensions except wl

INFO:2026-05-20,21:21:27 || radiative_transfer_engine.py:runSimulations() | Running any post-sim functions

INFO:2026-05-20,21:21:27 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:21:27 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:21:27 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:21:29 || isofit.py:run() | Beginning 100 inversions in 40 chunks using 4 cores

(Worker pid=4253) INFO:2026-05-20,21:21:37 ||| The coszen will be universal:   coszen=0.843794577691886

(Worker pid=4251) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
(Worker pid=4251)   return _methods._mean(a, axis=axis, dtype=dtype,
(Worker pid=4251) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
(Worker pid=4251)   ret = ret.dtype.type(ret / rcount)

(Worker pid=4251) INFO:2026-05-20,21:21:38 ||| Worker 1 completed 1/~25.0:: 4.0% complete

(Worker pid=4251) INFO:2026-05-20,21:21:38 ||| Worker at start location (0,6) completed 1/2

(Worker pid=4251) INFO:2026-05-20,21:21:42 ||| The coszen will be universal:   coszen=0.843794577691886 [repeated 44x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)

(Worker pid=4250) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice [repeated 3x across cluster]
(Worker pid=4250)   return _methods._mean(a, axis=axis, dtype=dtype, [repeated 3x across cluster]
(Worker pid=4250) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide [repeated 3x across cluster]
(Worker pid=4250)   ret = ret.dtype.type(ret / rcount) [repeated 3x across cluster]

(Worker pid=4252) INFO:2026-05-20,21:21:43 ||| Worker 3 completed 12/~25.0:: 48.0% complete [repeated 20x across cluster]

(Worker pid=4252) INFO:2026-05-20,21:21:42 ||| Worker at start location (3,7) completed 2/3 [repeated 16x across cluster]

INFO:2026-05-20,21:21:47 || isofit.py:run() | Pixel class: uniform_surface inversions complete.

INFO:2026-05-20,21:21:47 || isofit.py:run() | Running totals

INFO:2026-05-20,21:21:47 || isofit.py:run() | 18.4s total

INFO:2026-05-20,21:21:47 || isofit.py:run() | 5.4354 spectra/s

INFO:2026-05-20,21:21:47 || isofit.py:run() | 1.3588 spectra/s/core

INFO:2026-05-20,21:21:47 || isofit.py:run() | All Inversions complete.

INFO:2026-05-20,21:21:47 || isofit.py:run() | Final totals

INFO:2026-05-20,21:21:47 || isofit.py:run() | 18.4s total

INFO:2026-05-20,21:21:47 || isofit.py:run() | 5.4354 spectra/s

INFO:2026-05-20,21:21:47 || isofit.py:run() | 1.3588 spectra/s/core

INFO:2026-05-20,21:21:47 || apply_oe.py:apply_oe() | Full (non-aerosol) LUTs:

INFO:2026-05-20,21:21:47 || apply_oe.py:apply_oe() | Elevation: None

INFO:2026-05-20,21:21:47 || apply_oe.py:apply_oe() | To-sensor zenith: None

INFO:2026-05-20,21:21:47 || apply_oe.py:apply_oe() | To-sun zenith: None

INFO:2026-05-20,21:21:47 || apply_oe.py:apply_oe() | Relative to-sun azimuth: None

INFO:2026-05-20,21:21:47 || apply_oe.py:apply_oe() | H2O Vapor: [0.9001 1.0961 1.292  1.4879]

INFO:2026-05-20,21:21:47 || apply_oe.py:apply_oe() | Writing main configuration file.

INFO:2026-05-20,21:21:47 || template_construction.py:load_climatology() | Loading Climatology

INFO:2026-05-20,21:21:47 || template_construction.py:load_climatology() | Climatology Loaded.  Aerosol State Vector:
{'AOT550': {'bounds': [0.04209, 1.0], 'scale': 1, 'init': 0.137881, 'prior_sigma': 10.0, 'prior_mean': 0.137881}}
Aerosol LUT Grid:
{'AOT550': [0.04209, 0.137881, 0.233672, 0.329463, 0.425254, 0.521045, 0.6168359999999999, 0.712627, 0.808418, 0.9042089999999999, 1.0]}
Aerosol model path:/home/runner/work/isofit-tutorials/data/aerosol_model.txt

INFO:2026-05-20,21:21:47 || ini.py:toTemplate() | Saved converted json to: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/config/ang20170323t202244_isofit.json.tmpl

INFO:2026-05-20,21:21:47 || apply_oe.py:apply_oe() | Running ISOFIT with full LUT

WARNING:2026-05-20,21:21:47 || __init__.py:checkNumThreads() | 
******************************************************************************************
! Number of threads is greater than 1 (currently: 4), this may greatly impact performance
! Please set this the environment variables 'MKL_NUM_THREADS' and 'OMP_NUM_THREADS' to '1'
******************************************************************************************

INFO:2026-05-20,21:21:47 || configs.py:create_new_config() | Loading config file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/config/ang20170323t202244_isofit.json

(Worker pid=4253) INFO:2026-05-20,21:21:47 ||| The coszen will be universal:   coszen=0.843794577691886 [repeated 55x across cluster]INFO:2026-05-20,21:21:47 || configs.py:get_config_errors() | Checking config sections for configuration issues

INFO:2026-05-20,21:21:47 || configs.py:get_config_errors() | Configuration file checks complete, no errors found.

/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)

2026-05-20 21:21:48,239	INFO worker.py:1828 -- Calling ray.init() again after it has already been called.

INFO:2026-05-20,21:21:48 || isofit.py:run() | Running surfaces: uniform_surface

INFO:2026-05-20,21:21:48 || radiative_transfer_engine.py:__init__() | Loading from wavelength_file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/data/wavelengths.txt

INFO:2026-05-20,21:21:48 || radiative_transfer_engine.py:__init__() | No LUT store found, beginning initialization and simulations

INFO:2026-05-20,21:21:48 || radiative_transfer_engine.py:__init__() | Initializing LUT file

INFO:2026-05-20,21:21:48 || radiative_transfer_engine.py:runSimulations() | Running any pre-sim functions

INFO:2026-05-20,21:21:48 || sRTMnet.py:preSim() | Creating a simulator configuration

INFO:2026-05-20,21:21:48 || sRTMnet.py:preSim() | Building simulator and executing (6S)

INFO:2026-05-20,21:21:48 || radiative_transfer_engine.py:__init__() | No LUT store found, beginning initialization and simulations

INFO:2026-05-20,21:21:48 || radiative_transfer_engine.py:__init__() | Initializing LUT file

INFO:2026-05-20,21:21:48 || radiative_transfer_engine.py:runSimulations() | Running any pre-sim functions

INFO:2026-05-20,21:21:48 || radiative_transfer_engine.py:runSimulations() | Executing parallel simulations

INFO:2026-05-20,21:22:05 || common.py:__call__() |  11.36% simulations complete (elapsed: 0:00:16.157614, rate: 0:00:00.367218, eta: 0:02:25.418526)

INFO:2026-05-20,21:22:05 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:09 || common.py:__call__() |  20.45% simulations complete (elapsed: 0:00:20.944934, rate: 0:00:00.476021, eta: 0:01:23.779736)

INFO:2026-05-20,21:22:09 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:14 || common.py:__call__() |  31.82% simulations complete (elapsed: 0:00:25.377802, rate: 0:00:00.576768, eta: 0:00:59.214871)

INFO:2026-05-20,21:22:14 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:18 || common.py:__call__() |  40.91% simulations complete (elapsed: 0:00:29.997730, rate: 0:00:00.681767, eta: 0:00:44.996595)

INFO:2026-05-20,21:22:18 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:23 || common.py:__call__() |  50.00% simulations complete (elapsed: 0:00:34.547840, rate: 0:00:00.785178, eta: 0:00:34.547840)

INFO:2026-05-20,21:22:23 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:28 || common.py:__call__() |  61.36% simulations complete (elapsed: 0:00:39.214015, rate: 0:00:00.891228, eta: 0:00:26.142677)

INFO:2026-05-20,21:22:28 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:32 || common.py:__call__() |  70.45% simulations complete (elapsed: 0:00:44.035361, rate: 0:00:01.000804, eta: 0:00:18.872298)

INFO:2026-05-20,21:22:32 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:37 || common.py:__call__() |  81.82% simulations complete (elapsed: 0:00:48.524454, rate: 0:00:01.102828, eta: 0:00:12.131114)

INFO:2026-05-20,21:22:37 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:42 || common.py:__call__() |  90.91% simulations complete (elapsed: 0:00:53.193576, rate: 0:00:01.208945, eta: 0:00:05.910397)

INFO:2026-05-20,21:22:42 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:46 || common.py:__call__() | 100.00% simulations complete (elapsed: 0:00:57.532950, rate: 0:00:01.307567, eta: 0:00:00)

INFO:2026-05-20,21:22:46 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:22:46 || radiative_transfer_engine.py:runSimulations() | Running any post-sim functions

INFO:2026-05-20,21:22:47 || radiative_transfer_engine.py:runSimulations() | Saving post-sim data to index zero of all dimensions except wl

INFO:2026-05-20,21:22:47 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:22:47 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:22:47 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:22:47 || sRTMnet.py:preSim() | Interpolating simulator quantities to emulator size

INFO:2026-05-20,21:22:47 || sRTMnet.py:preSim() | Loading and predicting with emulator on 4 cores

INFO:2026-05-20,21:22:48 || radiative_transfer_engine.py:runSimulations() | Saving pre-sim data to index zero of all dimensions except wl

INFO:2026-05-20,21:22:48 || radiative_transfer_engine.py:runSimulations() | Running any post-sim functions

INFO:2026-05-20,21:22:48 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:22:48 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:22:48 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:22:49 || isofit.py:run() | Beginning 100 inversions in 40 chunks using 4 cores

(Worker pid=4250) INFO:2026-05-20,21:21:47 ||| Worker 0 completed 25/~25.0:: 100.0% complete [repeated 18x across cluster]
(Worker pid=4253) INFO:2026-05-20,21:21:47 ||| Worker at start location (9,9) completed 2/3 [repeated 22x across cluster]
(Worker pid=4799) INFO:2026-05-20,21:22:57 ||| The coszen will be universal:   coszen=0.843794577691886
(Worker pid=4798) INFO:2026-05-20,21:22:57 ||| The coszen will be universal:   coszen=0.843794577691886

(Worker pid=4798) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
(Worker pid=4798)   return _methods._mean(a, axis=axis, dtype=dtype,
(Worker pid=4798) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
(Worker pid=4798)   ret = ret.dtype.type(ret / rcount)

(Worker pid=4799) INFO:2026-05-20,21:23:02 ||| Worker 0 completed 8/~25.0:: 32.0% complete [repeated 14x across cluster]
(Worker pid=4800) INFO:2026-05-20,21:23:02 ||| Worker at start location (2,9) completed 1/2 [repeated 12x across cluster]
(Worker pid=4801) INFO:2026-05-20,21:23:02 ||| The coszen will be universal:   coszen=0.843794577691886 [repeated 35x across cluster]

(Worker pid=4800) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice [repeated 3x across cluster]
(Worker pid=4800)   return _methods._mean(a, axis=axis, dtype=dtype, [repeated 3x across cluster]
(Worker pid=4800) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide [repeated 3x across cluster]
(Worker pid=4800)   ret = ret.dtype.type(ret / rcount) [repeated 3x across cluster]

(Worker pid=4800) INFO:2026-05-20,21:23:07 ||| Worker 2 completed 14/~25.0:: 56.0% complete [repeated 9x across cluster]
(Worker pid=4799) INFO:2026-05-20,21:23:07 ||| Worker at start location (5,2) completed 1/2 [repeated 8x across cluster]
(Worker pid=4799) INFO:2026-05-20,21:23:07 ||| The coszen will be universal:   coszen=0.843794577691886 [repeated 21x across cluster]

INFO:2026-05-20,21:23:12 || isofit.py:run() | Pixel class: uniform_surface inversions complete.

INFO:2026-05-20,21:23:12 || isofit.py:run() | Running totals

INFO:2026-05-20,21:23:12 || isofit.py:run() | 22.84s total

INFO:2026-05-20,21:23:12 || isofit.py:run() | 4.3785 spectra/s

INFO:2026-05-20,21:23:12 || isofit.py:run() | 1.0946 spectra/s/core

INFO:2026-05-20,21:23:12 || isofit.py:run() | All Inversions complete.

INFO:2026-05-20,21:23:12 || isofit.py:run() | Final totals

INFO:2026-05-20,21:23:12 || isofit.py:run() | 22.84s total

INFO:2026-05-20,21:23:12 || isofit.py:run() | 4.3785 spectra/s

INFO:2026-05-20,21:23:12 || isofit.py:run() | 1.0946 spectra/s/core

INFO:2026-05-20,21:23:12 || apply_oe.py:apply_oe() | Done.

(Worker pid=4799) INFO:2026-05-20,21:23:11 ||| Worker 0 completed 23/~25.0:: 92.0% complete [repeated 16x across cluster]
(Worker pid=4799) INFO:2026-05-20,21:23:12 ||| Worker at start location (9,9) completed 2/3 [repeated 19x across cluster]
(Worker pid=4799) INFO:2026-05-20,21:23:12 ||| The coszen will be universal:   coszen=0.843794577691886 [repeated 42x across cluster]

We can examine both the inputs and outputs of Apply OE with this run call:

In [3]:

# Common imports
import os
from pathlib import Path

from spectral import envi
from matplotlib import pyplot as plt
import numpy as np

from isofit.core.common import envi_header
from isofit.data import env

# Common imports import os from pathlib import Path from spectral import envi from matplotlib import pyplot as plt import numpy as np from isofit.core.common import envi_header from isofit.data import env

In [4]:

# Load the input files
rdn_path = env.path("examples", "image_cube", "small", "data", "ang20170323t202244_rdn_7000-7010")
loc_path = env.path("examples", "image_cube", "small", "data", "ang20170323t202244_loc_7000-7010")
obs_path = env.path("examples", "image_cube", "small", "data", "ang20170323t202244_obs_7000-7010")

rdn = envi.open(envi_header(str(rdn_path)))
loc = envi.open(envi_header(str(loc_path)))
obs = envi.open(envi_header(str(obs_path)))

rdn_im = rdn.open_memmap(interleave='bip')
loc_im = loc.open_memmap(interleave='bip')
obs_im = obs.open_memmap(interleave='bip')

# Load the input files rdn_path = env.path("examples", "image_cube", "small", "data", "ang20170323t202244_rdn_7000-7010") loc_path = env.path("examples", "image_cube", "small", "data", "ang20170323t202244_loc_7000-7010") obs_path = env.path("examples", "image_cube", "small", "data", "ang20170323t202244_obs_7000-7010") rdn = envi.open(envi_header(str(rdn_path))) loc = envi.open(envi_header(str(loc_path))) obs = envi.open(envi_header(str(obs_path))) rdn_im = rdn.open_memmap(interleave='bip') loc_im = loc.open_memmap(interleave='bip') obs_im = obs.open_memmap(interleave='bip')

In [5]:

# Print the bands of the input files
print('Band names in the location file:')
[print(f"{i}") for i in loc.metadata['band names']]

print()
print('Band names in the observational geometry file:')
temp = [print(f"{i}") for i in obs.metadata['band names']]

# Print the bands of the input files print('Band names in the location file:') [print(f"{i}") for i in loc.metadata['band names']] print() print('Band names in the observational geometry file:') temp = [print(f"{i}") for i in obs.metadata['band names']]

Band names in the location file:
Longitude (WGS-84)
Latitude (WGS-84)
Elevation (m)

Band names in the observational geometry file:
Path length (m)
To-sensor azimuth (0 to 360 degrees cw from N)
To-sensor zenith (0 to 90 degrees from zenith)
To-sun azimuth (0 to 360 degrees cw from N)
To-sun zenith (0 to 90 degrees from zenith)
Solar phase
Slope
Aspect
Cosine(i)
UTC Time
Earth-sun distance (AU)

In [6]:

# Plot the input data
normalize = lambda x, vmin, vmax:  (x - vmin) / (vmax - vmin)
bands = [55, 35, 15]

fig, axs = plt.subplots(1, 3, sharex=True, sharey=True, figsize=(13, 4))
plot = axs[0].imshow(normalize(rdn_im[..., bands], 0, 15))
plot = axs[1].imshow(loc_im[..., 0])
plot = axs[2].imshow(obs_im[..., 4])

title = axs[0].set_title('Radiance (RGB)')
title = axs[1].set_title('Longitude (WGS-84)')
title = axs[2].set_title('Solar zenith angle (Deg)')

# Plot the input data normalize = lambda x, vmin, vmax: (x - vmin) / (vmax - vmin) bands = [55, 35, 15] fig, axs = plt.subplots(1, 3, sharex=True, sharey=True, figsize=(13, 4)) plot = axs[0].imshow(normalize(rdn_im[..., bands], 0, 15)) plot = axs[1].imshow(loc_im[..., 0]) plot = axs[2].imshow(obs_im[..., 4]) title = axs[0].set_title('Radiance (RGB)') title = axs[1].set_title('Longitude (WGS-84)') title = axs[2].set_title('Solar zenith angle (Deg)')

The input image doesn't look like much because this is just a 10x10 pixel example. However we see per-pixel spectral variation in the radiance RGBs, and systematic variation in the location and geometric variables.

In [7]:

# Load the output files
rfl_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_rfl")
state_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_state")
uncert_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_uncert")

rfl = envi.open(envi_header(str(rfl_path)))
state = envi.open(envi_header(str(state_path)))
uncert = envi.open(envi_header(str(uncert_path)))

rfl_im = rfl.open_memmap(interleave='bip')
state_im = state.open_memmap(interleave='bip')
uncert_im = uncert.open_memmap(interleave='bip')

print(f'Shape of the _rfl file: {rfl_im.shape}')
print(f'Shape of the _state file: {state_im.shape}')
print(f'Shape of the _uncert file: {uncert_im.shape}')

# Load the output files rfl_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_rfl") state_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_state") uncert_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_uncert") rfl = envi.open(envi_header(str(rfl_path))) state = envi.open(envi_header(str(state_path))) uncert = envi.open(envi_header(str(uncert_path))) rfl_im = rfl.open_memmap(interleave='bip') state_im = state.open_memmap(interleave='bip') uncert_im = uncert.open_memmap(interleave='bip') print(f'Shape of the _rfl file: {rfl_im.shape}') print(f'Shape of the _state file: {state_im.shape}') print(f'Shape of the _uncert file: {uncert_im.shape}')

Shape of the _rfl file: (10, 10, 425)
Shape of the _state file: (10, 10, 427)
Shape of the _uncert file: (10, 10, 427)

The difference between the _rfl file and the _state file is that the _rfl file only contains the solutions for surface reflectance variables. Here, the AVIRIS-NG image contains 425 wavelength bands. As a result, the _rfl contains 425 bands. The _state and _uncert files contain the surface reflectance solutions and uncertainty calculated as the standard deviation of the posterior distributions for the 425 wavelength bands and for non-reflectance statevector elements; here, aerosol optical depth (AOD) and water vapor (H2O).

In [8]:

# Plot the output data
normalize = lambda x, vmin, vmax:  (x - vmin) / (vmax - vmin)
bands = [55, 35, 15]

fig, axs = plt.subplots(2, 2, sharex=True, sharey=True, figsize=(8, 8))
axs = np.ravel(axs)
plot = axs[0].imshow(normalize(rfl_im[..., bands], 0, 0.3))
uncert = axs[1].imshow(uncert_im[..., 55])
aod = axs[2].imshow(state_im[..., -2])
h2o = axs[3].imshow(state_im[..., -1])

plt.colorbar(uncert)
plt.colorbar(aod)
plt.colorbar(h2o)

title = axs[0].set_title('Reflectance')
title = axs[1].set_title('Uncertainty (at 650 nm)')
title = axs[2].set_title('AOD')
title = axs[3].set_title('H2O')

# Plot the output data normalize = lambda x, vmin, vmax: (x - vmin) / (vmax - vmin) bands = [55, 35, 15] fig, axs = plt.subplots(2, 2, sharex=True, sharey=True, figsize=(8, 8)) axs = np.ravel(axs) plot = axs[0].imshow(normalize(rfl_im[..., bands], 0, 0.3)) uncert = axs[1].imshow(uncert_im[..., 55]) aod = axs[2].imshow(state_im[..., -2]) h2o = axs[3].imshow(state_im[..., -1]) plt.colorbar(uncert) plt.colorbar(aod) plt.colorbar(h2o) title = axs[0].set_title('Reflectance') title = axs[1].set_title('Uncertainty (at 650 nm)') title = axs[2].set_title('AOD') title = axs[3].set_title('H2O')

The Analytical Line¶

In the example above, we ran Isofit using the full optimal estimation (OE) on each pixel independently. Computationally, this amounts to iterating through each row-column pair to solve for the full state-vector (427 variables in the above case). However for large images, this demands long run-times, and ignores the principle that some state-vector elements, namely the atmospheric variables like AOD and H2O, should not vary from one pixel to another, but rather should be spatially smooth, and only vary over multi-pixel length scales Link to the relevent paper.

The analytical line and empirical line algorithms leverage the assumption of a spatially smooth atmosphere to decrease run times by a factor of 10. Currently, we suggest using the analytical line algorithm and not the empirical line algorithm.

See the following CLI call to run Apply OE with the analytical line algorithm:

  IC=$(isofit path examples)/image_cube/medium EX=$(isofit path examples)/image_cube/medium \
  isofit apply_oe \
  $IC/ang20170323t202244_rdn_7k-8k \
  $IC/ang20170323t202244_loc_7k-8k \
  $IC/ang20170323t202244_obs_7k-8k \
  $EX \
  ang \
  --surface_path $EX/configs/surface.json \
  --emulator_base $(isofit path srtmnet --key file) \
  --n_cores 10 \
  --presolve \
  --analytical_line \
  --segmentation_size 50 \
  --log_file $EX/log.txt

Most of the input parameters are identical to the per-pixel application above. However, we've added a --log_file, the --analytical_line flag, and a --segmentation_size. The --log_file points the program to write a text file to print logging statements during run time. The --analytical_line flag tells Isofit to use the analytical line algorithm.

A simple overview for the anylitical line algorithm:

1. All three input files are "segmented" into superpixel blocks using the SLIC algorithm. The --segmentation_size value sets the number of pixels that each superpixel contains.

2. At the superpixel resolution, Isofit solves for the OE solutions, which provides both surface and atmospheric state variables.

3. Atmospheric state variables are spatially interpolated to full image resolution. The spatial interpolation uses the Apply OE parameters --num_neighbors and --atm_sigma.

4. With a fixed atmosphere, we leverage a closed form solution for surface state elements that allows for a solution convergence in a single iteration.

In [9]:

# Alternatively, you can programatically call apply_oe. If you executed apply_oe via the command line above, do not run this
# We use the env object to retrieve the actual location of the ISOFIT extra dependencies instead of assuming where they could be
import os
import shutil

from isofit.utils.apply_oe import apply_oe
from isofit.data import env

output = env.path("examples", "image_cube", "medium")

# Cleanup any previous runs; comment this out if you want to preserve a previous run's output
if (o := output / "output").exists():
    shutil.rmtree(o)

apply_oe(
    input_radiance    = str(env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_rdn_7k-8k")),
    input_loc         = str(env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_loc_7k-8k")),
    input_obs         = str(env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_obs_7k-8k")),
    working_directory = str(output),
    sensor            = "ang",
    surface_path      = str(env.path("examples", "image_cube", "medium", "configs", "surface.json")),
    emulator_base     = str(env.path("srtmnet", key="srtmnet.file")),
    presolve          = True,
    n_cores           = os.cpu_count(),
    analytical_line   = True,
    segmentation_size = 50,
)

# Alternatively, you can programatically call apply_oe. If you executed apply_oe via the command line above, do not run this # We use the env object to retrieve the actual location of the ISOFIT extra dependencies instead of assuming where they could be import os import shutil from isofit.utils.apply_oe import apply_oe from isofit.data import env output = env.path("examples", "image_cube", "medium") # Cleanup any previous runs; comment this out if you want to preserve a previous run's output if (o := output / "output").exists(): shutil.rmtree(o) apply_oe( input_radiance = str(env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_rdn_7k-8k")), input_loc = str(env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_loc_7k-8k")), input_obs = str(env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_obs_7k-8k")), working_directory = str(output), sensor = "ang", surface_path = str(env.path("examples", "image_cube", "medium", "configs", "surface.json")), emulator_base = str(env.path("srtmnet", key="srtmnet.file")), presolve = True, n_cores = os.cpu_count(), analytical_line = True, segmentation_size = 50, )

INFO:2026-05-20,21:23:14 || apply_oe.py:apply_oe() | Checking input data files...

INFO:2026-05-20,21:23:14 || apply_oe.py:apply_oe() | ...Data file checks complete

INFO:2026-05-20,21:23:14 || apply_oe.py:apply_oe() | Setting up files and directories....

INFO:2026-05-20,21:23:14 || template_construction.py:__init__() | Flightline ID: ang20170323t202244

INFO:2026-05-20,21:23:14 || template_construction.py:__init__() | no noise path found, proceeding without

INFO:2026-05-20,21:23:14 || apply_oe.py:apply_oe() | ...file/directory setup complete

INFO:2026-05-20,21:23:14 || apply_oe.py:apply_oe() | Using inversion windows: [[350.0, 1360.0], [1410, 1800.0], [1970.0, 2500.0]]

INFO:2026-05-20,21:23:14 || template_construction.py:get_wavelengths() | No wavelength file provided. Obtaining wavelength grid from ENVI header of radiance cube.

INFO:2026-05-20,21:23:14 || template_construction.py:get_wavelengths() | Wavelength units of nm inferred...converting to microns

INFO:2026-05-20,21:23:14 || template_construction.py:check_surface_model() | No surface model provided. Build new one using given config file.

0 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

1 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']
2 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

3 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

4 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

5 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

6 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | Observation means:

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | Path (km): 20.427719754829514

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | To-sensor azimuth (deg): 70.76934723658717

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | To-sensor zenith (deg): 14.66271150290934

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | To-sun azimuth (deg): 197.26038313891075

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | To-sun zenith (deg): 32.53545303568223

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | Relative to-sun azimuth (deg): 126.4910359023236

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | Altitude (km): 19.82240248090924

INFO:2026-05-20,21:23:17 || apply_oe.py:apply_oe() | Segmenting...

7 ['/home/runner/work/isofit-tutorials/data/reflectance/surface_model_ucsb']

2026-05-20 21:23:22,601	INFO worker.py:2012 -- Started a local Ray instance.

(segment_chunk pid=5114) INFO:2026-05-20,21:23:26 ||| 0: starting

INFO:2026-05-20,21:23:27 || apply_oe.py:apply_oe() | Extracting /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_rdn

2026-05-20 21:23:27,231	INFO worker.py:1828 -- Calling ray.init() again after it has already been called.

(segment_chunk pid=5114) INFO:2026-05-20,21:23:27 ||| 0: completing
(extract_chunk pid=5114) INFO:2026-05-20,21:23:27 ||| Loaded ini from: /home/runner/.isofit/isofit.ini

INFO:2026-05-20,21:23:30 || apply_oe.py:apply_oe() | Extracting /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_obs

2026-05-20 21:23:30,628	INFO worker.py:1828 -- Calling ray.init() again after it has already been called.

INFO:2026-05-20,21:23:30 || apply_oe.py:apply_oe() | Extracting /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_loc

2026-05-20 21:23:30,754	INFO worker.py:1828 -- Calling ray.init() again after it has already been called.

INFO:2026-05-20,21:23:30 || apply_oe.py:apply_oe() | Skipping None, because is not a path.

INFO:2026-05-20,21:23:30 || apply_oe.py:apply_oe() | Skipping None, because is not a path.

INFO:2026-05-20,21:23:30 || apply_oe.py:apply_oe() | Pre-solve H2O grid: [0.2  0.76 1.32 1.88 2.44 2.99 3.55 4.11 4.67 5.23]

INFO:2026-05-20,21:23:30 || apply_oe.py:apply_oe() | Writing H2O pre-solve configuration file.

INFO:2026-05-20,21:23:30 || ini.py:toTemplate() | Saved converted json to: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/config/ang20170323t202244_h2o.json.tmpl

INFO:2026-05-20,21:23:30 || apply_oe.py:apply_oe() | Run ISOFIT initial guess

WARNING:2026-05-20,21:23:30 || __init__.py:checkNumThreads() | 
******************************************************************************************
! Number of threads is greater than 1 (currently: 4), this may greatly impact performance
! Please set this the environment variables 'MKL_NUM_THREADS' and 'OMP_NUM_THREADS' to '1'
******************************************************************************************

INFO:2026-05-20,21:23:30 || configs.py:create_new_config() | Loading config file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/config/ang20170323t202244_h2o.json

INFO:2026-05-20,21:23:30 || configs.py:get_config_errors() | Checking config sections for configuration issues

INFO:2026-05-20,21:23:30 || configs.py:get_config_errors() | Configuration file checks complete, no errors found.

/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)

2026-05-20 21:23:31,150	INFO worker.py:1828 -- Calling ray.init() again after it has already been called.

INFO:2026-05-20,21:23:31 || isofit.py:run() | Running surfaces: uniform_surface

INFO:2026-05-20,21:23:31 || radiative_transfer_engine.py:__init__() | Loading from wavelength_file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/data/wavelengths.txt

INFO:2026-05-20,21:23:31 || radiative_transfer_engine.py:__init__() | No LUT store found, beginning initialization and simulations

INFO:2026-05-20,21:23:31 || radiative_transfer_engine.py:__init__() | Initializing LUT file

INFO:2026-05-20,21:23:31 || radiative_transfer_engine.py:runSimulations() | Running any pre-sim functions

INFO:2026-05-20,21:23:31 || sRTMnet.py:preSim() | Creating a simulator configuration

INFO:2026-05-20,21:23:31 || sRTMnet.py:preSim() | Building simulator and executing (6S)

INFO:2026-05-20,21:23:31 || radiative_transfer_engine.py:__init__() | No LUT store found, beginning initialization and simulations

INFO:2026-05-20,21:23:31 || radiative_transfer_engine.py:__init__() | Initializing LUT file

INFO:2026-05-20,21:23:31 || radiative_transfer_engine.py:runSimulations() | Running any pre-sim functions

INFO:2026-05-20,21:23:31 || radiative_transfer_engine.py:runSimulations() | Executing parallel simulations

INFO:2026-05-20,21:23:35 || common.py:__call__() |  20.00% simulations complete (elapsed: 0:00:03.868539, rate: 0:00:00.386854, eta: 0:00:34.816851)

INFO:2026-05-20,21:23:35 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:35 || common.py:__call__() |  30.00% simulations complete (elapsed: 0:00:04.166636, rate: 0:00:00.416664, eta: 0:00:16.666544)

INFO:2026-05-20,21:23:35 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:36 || common.py:__call__() |  40.00% simulations complete (elapsed: 0:00:04.463955, rate: 0:00:00.446396, eta: 0:00:10.415895)

INFO:2026-05-20,21:23:36 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:39 || common.py:__call__() |  50.00% simulations complete (elapsed: 0:00:07.656534, rate: 0:00:00.765653, eta: 0:00:11.484801)

INFO:2026-05-20,21:23:39 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:39 || common.py:__call__() |  60.00% simulations complete (elapsed: 0:00:07.952498, rate: 0:00:00.795250, eta: 0:00:07.952498)

INFO:2026-05-20,21:23:39 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:39 || common.py:__call__() |  70.00% simulations complete (elapsed: 0:00:08.247709, rate: 0:00:00.824771, eta: 0:00:05.498473)

INFO:2026-05-20,21:23:39 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:40 || common.py:__call__() |  80.00% simulations complete (elapsed: 0:00:08.540329, rate: 0:00:00.854033, eta: 0:00:03.660141)

INFO:2026-05-20,21:23:40 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:41 || common.py:__call__() |  90.00% simulations complete (elapsed: 0:00:10.130464, rate: 0:00:01.013046, eta: 0:00:02.532616)

INFO:2026-05-20,21:23:41 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:42 || common.py:__call__() | 100.00% simulations complete (elapsed: 0:00:10.368097, rate: 0:00:01.036810, eta: 0:00:01.152011)

INFO:2026-05-20,21:23:42 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:23:42 || radiative_transfer_engine.py:runSimulations() | Running any post-sim functions

INFO:2026-05-20,21:23:42 || radiative_transfer_engine.py:runSimulations() | Saving post-sim data to index zero of all dimensions except wl

INFO:2026-05-20,21:23:43 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:23:43 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:23:43 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:23:43 || sRTMnet.py:preSim() | Interpolating simulator quantities to emulator size

INFO:2026-05-20,21:23:43 || sRTMnet.py:preSim() | Loading and predicting with emulator on 4 cores

INFO:2026-05-20,21:23:43 || radiative_transfer_engine.py:runSimulations() | Saving pre-sim data to index zero of all dimensions except wl

INFO:2026-05-20,21:23:44 || radiative_transfer_engine.py:runSimulations() | Running any post-sim functions

INFO:2026-05-20,21:23:44 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:23:44 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:23:44 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:23:44 || isofit.py:run() | Beginning 1998 inversions in 40 chunks using 4 cores

(Worker pid=5446) INFO:2026-05-20,21:23:52 ||| The coszen will be universal:   coszen=0.8430588183136042
(extract_chunk pid=5113) INFO:2026-05-20,21:23:30 ||| 0: starting [repeated 15x across cluster]
(segment_chunk pid=5113) INFO:2026-05-20,21:23:27 ||| 768: completing [repeated 3x across cluster]
(extract_chunk pid=5113) INFO:2026-05-20,21:23:27 ||| Loaded ini from: /home/runner/.isofit/isofit.ini [repeated 3x across cluster]

(Worker pid=5446) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
(Worker pid=5446)   return _methods._mean(a, axis=axis, dtype=dtype,
(Worker pid=5446) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
(Worker pid=5446)   ret = ret.dtype.type(ret / rcount)
(Worker pid=5446) INFO:2026-05-20,21:23:52 ||| Worker 2 completed 1/~499.0:: 0.2% complete

(Worker pid=5445) INFO:2026-05-20,21:23:57 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 48x across cluster]

(Worker pid=5445) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice [repeated 3x across cluster]
(Worker pid=5445)   return _methods._mean(a, axis=axis, dtype=dtype, [repeated 3x across cluster]
(Worker pid=5445) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide [repeated 3x across cluster]
(Worker pid=5445)   ret = ret.dtype.type(ret / rcount) [repeated 3x across cluster]
(Worker pid=5448) INFO:2026-05-20,21:23:53 ||| Worker 3 completed 1/~499.0:: 0.2% complete [repeated 3x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:24:02 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 50x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:24:07 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 48x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:24:09 ||| Worker at start location (101,0) completed 50/51

(Worker pid=5446) INFO:2026-05-20,21:24:09 ||| Worker 2 completed 52/~499.0:: 10.42% complete

(Worker pid=5448) INFO:2026-05-20,21:24:12 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 38x across cluster]

(Worker pid=5445) INFO:2026-05-20,21:24:15 ||| Worker at start location (152,0) completed 50/51

(Worker pid=5445) INFO:2026-05-20,21:24:15 ||| Worker 1 completed 52/~499.0:: 10.42% complete

(Worker pid=5445) INFO:2026-05-20,21:24:17 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 41x across cluster]

(Worker pid=5448) INFO:2026-05-20,21:24:17 ||| Worker at start location (50,0) completed 50/51 [repeated 2x across cluster]

(Worker pid=5448) INFO:2026-05-20,21:24:18 ||| Worker 3 completed 52/~499.0:: 10.42% complete [repeated 2x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:24:22 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 29x across cluster]

(Worker pid=5448) INFO:2026-05-20,21:24:27 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 33x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:24:32 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 31x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:24:37 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 33x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:24:42 ||| Worker at start location (255,0) completed 51/52

(Worker pid=5447) INFO:2026-05-20,21:24:42 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 31x across cluster]
(Worker pid=5446) INFO:2026-05-20,21:24:42 ||| Worker 2 completed 104/~499.0:: 20.84% complete

(Worker pid=5448) INFO:2026-05-20,21:24:44 ||| Worker at start location (408,0) completed 50/51

(Worker pid=5448) INFO:2026-05-20,21:24:47 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 35x across cluster]
(Worker pid=5448) INFO:2026-05-20,21:24:45 ||| Worker 3 completed 103/~499.0:: 20.64% complete

(Worker pid=5445) INFO:2026-05-20,21:24:49 ||| Worker at start location (306,0) completed 50/51

(Worker pid=5445) INFO:2026-05-20,21:24:49 ||| Worker 1 completed 103/~499.0:: 20.64% complete

(Worker pid=5446) INFO:2026-05-20,21:24:52 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 45x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:24:52 ||| Worker at start location (357,0) completed 50/51

(Worker pid=5447) INFO:2026-05-20,21:24:53 ||| Worker 0 completed 103/~499.0:: 20.64% complete

(Worker pid=5445) INFO:2026-05-20,21:24:57 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 41x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:25:03 ||| Worker at start location (460,0) completed 51/52
(Worker pid=5446) INFO:2026-05-20,21:25:03 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 54x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:25:03 ||| Worker 2 completed 156/~499.0:: 31.26% complete

(Worker pid=5448) INFO:2026-05-20,21:25:07 ||| Worker at start location (511,0) completed 50/51
(Worker pid=5447) INFO:2026-05-20,21:25:08 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 46x across cluster]

(Worker pid=5448) INFO:2026-05-20,21:25:07 ||| Worker 3 completed 154/~499.0:: 30.86% complete

(Worker pid=5447) INFO:2026-05-20,21:25:10 ||| Worker at start location (613,0) completed 50/51

(Worker pid=5447) INFO:2026-05-20,21:25:10 ||| Worker 0 completed 154/~499.0:: 30.86% complete

(Worker pid=5445) INFO:2026-05-20,21:25:13 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 81x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:25:16 ||| Worker at start location (665,0) completed 51/52

(Worker pid=5446) INFO:2026-05-20,21:25:16 ||| Worker 2 completed 208/~499.0:: 41.68% complete

(Worker pid=5447) INFO:2026-05-20,21:25:18 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 88x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:21 ||| Worker at start location (767,0) completed 50/51 [repeated 3x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:21 ||| Worker 0 completed 205/~499.0:: 41.08% complete [repeated 3x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:23 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 100x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:25:26 ||| Worker at start location (818,0) completed 50/51

(Worker pid=5446) INFO:2026-05-20,21:25:26 ||| Worker 2 completed 259/~499.0:: 51.9% complete

(Worker pid=5447) INFO:2026-05-20,21:25:28 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 100x across cluster]

(Worker pid=5448) INFO:2026-05-20,21:25:28 ||| Worker at start location (921,0) completed 51/52 [repeated 2x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:31 ||| Worker 0 completed 256/~499.0:: 51.3% complete [repeated 3x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:25:33 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 100x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:31 ||| Worker at start location (972,0) completed 50/51

(Worker pid=5446) INFO:2026-05-20,21:25:36 ||| Worker at start location (1023,0) completed 50/51

(Worker pid=5446) INFO:2026-05-20,21:25:37 ||| Worker 2 completed 310/~499.0:: 62.12% complete

(Worker pid=5445) INFO:2026-05-20,21:25:38 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 100x across cluster]

(Worker pid=5448) INFO:2026-05-20,21:25:38 ||| Worker at start location (1126,0) completed 51/52 [repeated 2x across cluster]

(Worker pid=5448) INFO:2026-05-20,21:25:39 ||| Worker 3 completed 309/~499.0:: 61.92% complete [repeated 2x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:43 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 100x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:42 ||| Worker at start location (1177,0) completed 50/51

(Worker pid=5447) INFO:2026-05-20,21:25:42 ||| Worker 0 completed 307/~499.0:: 61.52% complete

(Worker pid=5446) INFO:2026-05-20,21:25:47 ||| Worker at start location (1228,0) completed 50/51

(Worker pid=5445) INFO:2026-05-20,21:25:47 ||| Worker 1 completed 307/~499.0:: 61.52% complete

(Worker pid=5447) INFO:2026-05-20,21:25:48 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 101x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:52 ||| Worker at start location (1382,0) completed 50/51 [repeated 3x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:25:52 ||| Worker 0 completed 358/~499.0:: 71.74% complete [repeated 3x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:25:53 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 101x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:25:57 ||| Worker at start location (1433,0) completed 50/51
(Worker pid=5446) INFO:2026-05-20,21:25:58 ||| Worker 2 completed 412/~499.0:: 82.57% complete

(Worker pid=5447) INFO:2026-05-20,21:25:58 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 98x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:26:02 ||| Worker at start location (1587,0) completed 51/52 [repeated 3x across cluster]
(Worker pid=5447) INFO:2026-05-20,21:26:02 ||| Worker 0 completed 410/~499.0:: 82.16% complete [repeated 3x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:26:03 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 101x across cluster]

(Worker pid=5446) INFO:2026-05-20,21:26:08 ||| Worker at start location (1638,0) completed 50/51

(Worker pid=5446) INFO:2026-05-20,21:26:08 ||| Worker 2 completed 463/~499.0:: 92.79% complete

(Worker pid=5445) INFO:2026-05-20,21:26:08 ||| Worker at start location (1689,0) completed 50/51
(Worker pid=5445) INFO:2026-05-20,21:26:08 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 97x across cluster]

(Worker pid=5448) INFO:2026-05-20,21:26:10 ||| Worker 3 completed 463/~499.0:: 92.79% complete [repeated 2x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:26:13 ||| Worker at start location (1792,0) completed 51/52 [repeated 2x across cluster]
(Worker pid=5445) INFO:2026-05-20,21:26:14 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 98x across cluster]

(Worker pid=5447) INFO:2026-05-20,21:26:13 ||| Worker 0 completed 462/~499.0:: 92.59% complete

(Worker pid=5446) INFO:2026-05-20,21:26:18 ||| Worker at start location (1843,0) completed 50/51
(Worker pid=5447) INFO:2026-05-20,21:26:19 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 97x across cluster]

(Worker pid=5445) INFO:2026-05-20,21:26:19 ||| Worker at start location (1894,0) completed 50/51

INFO:2026-05-20,21:26:22 || isofit.py:run() | Pixel class: uniform_surface inversions complete.

INFO:2026-05-20,21:26:22 || isofit.py:run() | Running totals

INFO:2026-05-20,21:26:22 || isofit.py:run() | 157.84s total

INFO:2026-05-20,21:26:22 || isofit.py:run() | 12.6587 spectra/s

INFO:2026-05-20,21:26:22 || isofit.py:run() | 3.1647 spectra/s/core

INFO:2026-05-20,21:26:22 || isofit.py:run() | All Inversions complete.

INFO:2026-05-20,21:26:22 || isofit.py:run() | Final totals

INFO:2026-05-20,21:26:22 || isofit.py:run() | 157.84s total

INFO:2026-05-20,21:26:22 || isofit.py:run() | 12.6587 spectra/s

INFO:2026-05-20,21:26:22 || isofit.py:run() | 3.1647 spectra/s/core

INFO:2026-05-20,21:26:22 || apply_oe.py:apply_oe() | Full (non-aerosol) LUTs:

INFO:2026-05-20,21:26:22 || apply_oe.py:apply_oe() | Elevation: None

INFO:2026-05-20,21:26:22 || apply_oe.py:apply_oe() | To-sensor zenith: [11.5786 17.7236]

INFO:2026-05-20,21:26:22 || apply_oe.py:apply_oe() | To-sun zenith: None

INFO:2026-05-20,21:26:22 || apply_oe.py:apply_oe() | Relative to-sun azimuth: None

INFO:2026-05-20,21:26:22 || apply_oe.py:apply_oe() | H2O Vapor: [1.0156 1.2242 1.4328]

INFO:2026-05-20,21:26:22 || apply_oe.py:apply_oe() | Writing main configuration file.

INFO:2026-05-20,21:26:22 || template_construction.py:load_climatology() | Loading Climatology

INFO:2026-05-20,21:26:22 || template_construction.py:load_climatology() | Climatology Loaded.  Aerosol State Vector:
{'AOT550': {'bounds': [0.04209, 1.0], 'scale': 1, 'init': 0.137881, 'prior_sigma': 10.0, 'prior_mean': 0.137881}}
Aerosol LUT Grid:
{'AOT550': [0.04209, 0.137881, 0.233672, 0.329463, 0.425254, 0.521045, 0.6168359999999999, 0.712627, 0.808418, 0.9042089999999999, 1.0]}
Aerosol model path:/home/runner/work/isofit-tutorials/data/aerosol_model.txt

INFO:2026-05-20,21:26:22 || ini.py:toTemplate() | Saved converted json to: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/config/ang20170323t202244_isofit.json.tmpl

INFO:2026-05-20,21:26:22 || apply_oe.py:apply_oe() | Running ISOFIT with full LUT

WARNING:2026-05-20,21:26:22 || __init__.py:checkNumThreads() | 
******************************************************************************************
! Number of threads is greater than 1 (currently: 4), this may greatly impact performance
! Please set this the environment variables 'MKL_NUM_THREADS' and 'OMP_NUM_THREADS' to '1'
******************************************************************************************

INFO:2026-05-20,21:26:22 || configs.py:create_new_config() | Loading config file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/config/ang20170323t202244_isofit.json

INFO:2026-05-20,21:26:22 || configs.py:get_config_errors() | Checking config sections for configuration issues

INFO:2026-05-20,21:26:22 || configs.py:get_config_errors() | Configuration file checks complete, no errors found.

/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)

2026-05-20 21:26:22,847	INFO worker.py:1828 -- Calling ray.init() again after it has already been called.

INFO:2026-05-20,21:26:22 || isofit.py:run() | Running surfaces: uniform_surface

INFO:2026-05-20,21:26:22 || radiative_transfer_engine.py:__init__() | Loading from wavelength_file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/data/wavelengths.txt

INFO:2026-05-20,21:26:22 || radiative_transfer_engine.py:__init__() | No LUT store found, beginning initialization and simulations

INFO:2026-05-20,21:26:22 || radiative_transfer_engine.py:__init__() | Initializing LUT file

INFO:2026-05-20,21:26:23 || radiative_transfer_engine.py:runSimulations() | Running any pre-sim functions

INFO:2026-05-20,21:26:23 || sRTMnet.py:preSim() | Creating a simulator configuration

INFO:2026-05-20,21:26:23 || sRTMnet.py:preSim() | Building simulator and executing (6S)

INFO:2026-05-20,21:26:23 || radiative_transfer_engine.py:__init__() | No LUT store found, beginning initialization and simulations

INFO:2026-05-20,21:26:23 || radiative_transfer_engine.py:__init__() | Initializing LUT file

INFO:2026-05-20,21:26:23 || radiative_transfer_engine.py:runSimulations() | Running any pre-sim functions

INFO:2026-05-20,21:26:23 || radiative_transfer_engine.py:runSimulations() | Executing parallel simulations

INFO:2026-05-20,21:26:40 || common.py:__call__() |  10.61% simulations complete (elapsed: 0:00:16.669650, rate: 0:00:00.252570, eta: 0:02:30.026850)

INFO:2026-05-20,21:26:40 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:26:49 || common.py:__call__() |  21.21% simulations complete (elapsed: 0:00:25.633727, rate: 0:00:00.388390, eta: 0:01:42.534908)

INFO:2026-05-20,21:26:49 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:26:54 || common.py:__call__() |  30.30% simulations complete (elapsed: 0:00:30.734148, rate: 0:00:00.465669, eta: 0:01:11.713012)

INFO:2026-05-20,21:26:54 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:27:02 || common.py:__call__() |  40.91% simulations complete (elapsed: 0:00:38.999716, rate: 0:00:00.590905, eta: 0:00:58.499574)

INFO:2026-05-20,21:27:02 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:27:10 || common.py:__call__() |  50.00% simulations complete (elapsed: 0:00:47.088968, rate: 0:00:00.713469, eta: 0:00:47.088968)

INFO:2026-05-20,21:27:10 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:27:16 || common.py:__call__() |  60.61% simulations complete (elapsed: 0:00:52.762413, rate: 0:00:00.799430, eta: 0:00:35.174942)

INFO:2026-05-20,21:27:16 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:27:25 || common.py:__call__() |  71.21% simulations complete (elapsed: 0:01:01.647081, rate: 0:00:00.934047, eta: 0:00:26.420178)

INFO:2026-05-20,21:27:25 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:27:32 || common.py:__call__() |  80.30% simulations complete (elapsed: 0:01:09.083190, rate: 0:00:01.046715, eta: 0:00:17.270798)

INFO:2026-05-20,21:27:32 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:27:39 || common.py:__call__() |  90.91% simulations complete (elapsed: 0:01:15.510939, rate: 0:00:01.144105, eta: 0:00:08.390104)

INFO:2026-05-20,21:27:39 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:27:45 || common.py:__call__() | 100.00% simulations complete (elapsed: 0:01:22.287628, rate: 0:00:01.246782, eta: 0:00:00)

INFO:2026-05-20,21:27:45 || radiative_transfer_engine.py:runSimulations() | Flushing netCDF to disk

INFO:2026-05-20,21:27:46 || radiative_transfer_engine.py:runSimulations() | Running any post-sim functions

INFO:2026-05-20,21:27:46 || radiative_transfer_engine.py:runSimulations() | Saving post-sim data to index zero of all dimensions except wl

INFO:2026-05-20,21:27:46 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:27:46 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:27:46 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:27:46 || sRTMnet.py:preSim() | Interpolating simulator quantities to emulator size

INFO:2026-05-20,21:27:46 || sRTMnet.py:preSim() | Loading and predicting with emulator on 4 cores

INFO:2026-05-20,21:27:48 || radiative_transfer_engine.py:runSimulations() | Saving pre-sim data to index zero of all dimensions except wl

INFO:2026-05-20,21:27:48 || radiative_transfer_engine.py:runSimulations() | Running any post-sim functions

INFO:2026-05-20,21:27:48 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:27:48 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:27:48 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:27:49 || isofit.py:run() | Beginning 1998 inversions in 40 chunks using 4 cores

(Worker pid=6086) INFO:2026-05-20,21:27:57 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 33x across cluster]
(Worker pid=5447) INFO:2026-05-20,21:26:22 ||| Worker at start location (1997,0) completed 51/52 [repeated 2x across cluster]

(Worker pid=6086) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
(Worker pid=6086)   return _methods._mean(a, axis=axis, dtype=dtype,
(Worker pid=6086) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
(Worker pid=6086)   ret = ret.dtype.type(ret / rcount)

(Worker pid=6086) INFO:2026-05-20,21:27:59 ||| Worker 0 completed 1/~499.0:: 0.2% complete

(Worker pid=6083) INFO:2026-05-20,21:28:02 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 23x across cluster]

(Worker pid=6083) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice [repeated 3x across cluster]
(Worker pid=6083)   return _methods._mean(a, axis=axis, dtype=dtype, [repeated 3x across cluster]
(Worker pid=6083) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide [repeated 3x across cluster]
(Worker pid=6083)   ret = ret.dtype.type(ret / rcount) [repeated 3x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:28:00 ||| Worker 3 completed 1/~499.0:: 0.2% complete [repeated 3x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:28:07 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 37x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:28:12 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 34x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:28:17 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 33x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:28:23 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 35x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:28:25 ||| Worker at start location (101,0) completed 50/51

(Worker pid=6083) INFO:2026-05-20,21:28:26 ||| Worker 2 completed 52/~499.0:: 10.42% complete

(Worker pid=6083) INFO:2026-05-20,21:28:28 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 32x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:28:29 ||| Worker at start location (203,0) completed 50/51

(Worker pid=6084) INFO:2026-05-20,21:28:31 ||| Worker at start location (152,0) completed 50/51
(Worker pid=6086) INFO:2026-05-20,21:28:30 ||| Worker 0 completed 52/~499.0:: 10.42% complete

(Worker pid=6084) INFO:2026-05-20,21:28:32 ||| Worker 1 completed 52/~499.0:: 10.42% complete

(Worker pid=6084) INFO:2026-05-20,21:28:33 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 27x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:28:35 ||| Worker at start location (50,0) completed 50/51

(Worker pid=6085) INFO:2026-05-20,21:28:36 ||| Worker 3 completed 52/~499.0:: 10.42% complete

(Worker pid=6084) INFO:2026-05-20,21:28:38 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 23x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:28:44 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 29x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:28:49 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 25x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:28:54 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 23x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:28:59 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 23x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:29:04 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 22x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:29:08 ||| Worker at start location (255,0) completed 51/52

(Worker pid=6083) INFO:2026-05-20,21:29:08 ||| Worker 2 completed 104/~499.0:: 20.84% complete

(Worker pid=6084) INFO:2026-05-20,21:29:09 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 25x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:29:12 ||| Worker at start location (408,0) completed 50/51

(Worker pid=6085) INFO:2026-05-20,21:29:13 ||| Worker 3 completed 103/~499.0:: 20.64% complete

(Worker pid=6084) INFO:2026-05-20,21:29:14 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 29x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:29:15 ||| Worker at start location (306,0) completed 50/51

(Worker pid=6086) INFO:2026-05-20,21:29:15 ||| Worker 0 completed 103/~499.0:: 20.64% complete

(Worker pid=6085) INFO:2026-05-20,21:29:19 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 31x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:29:16 ||| Worker at start location (357,0) completed 50/51

(Worker pid=6084) INFO:2026-05-20,21:29:17 ||| Worker 1 completed 103/~499.0:: 20.64% complete

(Worker pid=6086) INFO:2026-05-20,21:29:25 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 29x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:29:30 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 34x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:29:35 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 33x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:29:39 ||| Worker at start location (460,0) completed 51/52

(Worker pid=6083) INFO:2026-05-20,21:29:40 ||| Worker 2 completed 156/~499.0:: 31.26% complete
(Worker pid=6085) INFO:2026-05-20,21:29:40 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 36x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:29:44 ||| Worker at start location (613,0) completed 50/51

(Worker pid=6084) INFO:2026-05-20,21:29:44 ||| Worker 1 completed 154/~499.0:: 30.86% complete
(Worker pid=6084) INFO:2026-05-20,21:29:45 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 39x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:29:45 ||| Worker at start location (511,0) completed 50/51

(Worker pid=6085) INFO:2026-05-20,21:29:46 ||| Worker 3 completed 154/~499.0:: 30.86% complete

(Worker pid=6086) INFO:2026-05-20,21:29:50 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 46x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:29:51 ||| Worker at start location (562,0) completed 50/51

(Worker pid=6086) INFO:2026-05-20,21:29:52 ||| Worker 0 completed 154/~499.0:: 30.86% complete

(Worker pid=6086) INFO:2026-05-20,21:29:55 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 52x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:30:00 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 56x across cluster]
(Worker pid=6083) INFO:2026-05-20,21:30:00 ||| Worker at start location (665,0) completed 51/52

(Worker pid=6083) INFO:2026-05-20,21:30:01 ||| Worker 2 completed 208/~499.0:: 41.68% complete

(Worker pid=6085) INFO:2026-05-20,21:30:05 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 56x across cluster]
(Worker pid=6085) INFO:2026-05-20,21:30:04 ||| Worker at start location (767,0) completed 50/51 [repeated 2x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:30:04 ||| Worker 3 completed 205/~499.0:: 41.08% complete [repeated 2x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:30:11 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:30:10 ||| Worker at start location (818,0) completed 50/51

(Worker pid=6086) INFO:2026-05-20,21:30:11 ||| Worker 0 completed 205/~499.0:: 41.08% complete

(Worker pid=6085) INFO:2026-05-20,21:30:16 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 55x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:30:19 ||| Worker at start location (869,0) completed 50/51

(Worker pid=6083) INFO:2026-05-20,21:30:19 ||| Worker 2 completed 259/~499.0:: 51.9% complete

(Worker pid=6083) INFO:2026-05-20,21:30:21 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:30:22 ||| Worker at start location (972,0) completed 50/51 [repeated 2x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:30:23 ||| Worker 3 completed 256/~499.0:: 51.3% complete [repeated 2x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:30:26 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 56x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:30:29 ||| Worker at start location (1023,0) completed 50/51

(Worker pid=6086) INFO:2026-05-20,21:30:29 ||| Worker 0 completed 256/~499.0:: 51.3% complete

(Worker pid=6083) INFO:2026-05-20,21:30:31 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:30:36 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 56x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:30:37 ||| Worker at start location (1074,0) completed 50/51

(Worker pid=6083) INFO:2026-05-20,21:30:37 ||| Worker 2 completed 310/~499.0:: 62.12% complete

(Worker pid=6083) INFO:2026-05-20,21:30:41 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 56x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:30:41 ||| Worker at start location (1177,0) completed 50/51 [repeated 2x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:30:41 ||| Worker 3 completed 307/~499.0:: 61.52% complete [repeated 2x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:30:46 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 56x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:30:49 ||| Worker at start location (1228,0) completed 50/51

(Worker pid=6086) INFO:2026-05-20,21:30:49 ||| Worker 0 completed 307/~499.0:: 61.52% complete

(Worker pid=6084) INFO:2026-05-20,21:30:51 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:30:55 ||| Worker at start location (1279,0) completed 50/51

(Worker pid=6083) INFO:2026-05-20,21:30:55 ||| Worker 2 completed 361/~499.0:: 72.34% complete

(Worker pid=6086) INFO:2026-05-20,21:30:56 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:30:59 ||| Worker at start location (1331,0) completed 51/52 [repeated 2x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:30:59 ||| Worker 1 completed 361/~499.0:: 72.34% complete [repeated 2x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:31:01 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:31:07 ||| Worker at start location (1433,0) completed 50/51
(Worker pid=6086) INFO:2026-05-20,21:31:07 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 59x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:31:07 ||| Worker 0 completed 358/~499.0:: 71.74% complete

(Worker pid=6085) INFO:2026-05-20,21:31:12 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 58x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:31:13 ||| Worker at start location (1484,0) completed 50/51

(Worker pid=6083) INFO:2026-05-20,21:31:14 ||| Worker 2 completed 412/~499.0:: 82.57% complete

(Worker pid=6086) INFO:2026-05-20,21:31:17 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:31:17 ||| Worker at start location (1587,0) completed 51/52 [repeated 2x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:31:18 ||| Worker 1 completed 413/~499.0:: 82.77% complete [repeated 2x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:31:22 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:31:24 ||| Worker at start location (1638,0) completed 50/51

(Worker pid=6086) INFO:2026-05-20,21:31:25 ||| Worker 0 completed 409/~499.0:: 81.96% complete

(Worker pid=6086) INFO:2026-05-20,21:31:27 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6083) INFO:2026-05-20,21:31:32 ||| Worker at start location (1689,0) completed 50/51

(Worker pid=6084) INFO:2026-05-20,21:31:32 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]
(Worker pid=6083) INFO:2026-05-20,21:31:32 ||| Worker 2 completed 463/~499.0:: 92.79% complete

(Worker pid=6084) INFO:2026-05-20,21:31:36 ||| Worker at start location (1792,0) completed 51/52 [repeated 2x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:31:37 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]
(Worker pid=6084) INFO:2026-05-20,21:31:36 ||| Worker 1 completed 465/~499.0:: 93.19% complete [repeated 2x across cluster]

(Worker pid=6085) INFO:2026-05-20,21:31:42 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 57x across cluster]

(Worker pid=6086) INFO:2026-05-20,21:31:43 ||| Worker at start location (1843,0) completed 50/51

(Worker pid=6084) INFO:2026-05-20,21:31:48 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 52x across cluster]

(Worker pid=6084) INFO:2026-05-20,21:31:50 ||| Worker at start location (1997,0) completed 51/52

INFO:2026-05-20,21:31:52 || isofit.py:run() | Pixel class: uniform_surface inversions complete.

INFO:2026-05-20,21:31:52 || isofit.py:run() | Running totals

INFO:2026-05-20,21:31:52 || isofit.py:run() | 242.72s total

INFO:2026-05-20,21:31:52 || isofit.py:run() | 8.2316 spectra/s

INFO:2026-05-20,21:31:52 || isofit.py:run() | 2.0579 spectra/s/core

INFO:2026-05-20,21:31:52 || isofit.py:run() | All Inversions complete.

INFO:2026-05-20,21:31:52 || isofit.py:run() | Final totals

INFO:2026-05-20,21:31:52 || isofit.py:run() | 242.72s total

INFO:2026-05-20,21:31:52 || isofit.py:run() | 8.2316 spectra/s

INFO:2026-05-20,21:31:52 || isofit.py:run() | 2.0579 spectra/s/core

INFO:2026-05-20,21:31:52 || apply_oe.py:apply_oe() | Analytical line inference

INFO:2026-05-20,21:31:52 || configs.py:create_new_config() | Loading config file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/config/ang20170323t202244_isofit.json

/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)

2026-05-20 21:31:52,571	INFO worker.py:1828 -- Calling ray.init() again after it has already been called.

INFO:2026-05-20,21:31:52 || atm_interpolation.py:atm_interpolation() | Beginning atmospheric interpolation 4 cores

(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.1104429821071497e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0991187168708807e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.094087694021587e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0889319959283916e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0682732967169464e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0615913951305175e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0306169133730392e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0306170306875678e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9976791930123204e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9726421839469346e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9371198852183326e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9250013699051746e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.897007115295885e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.972642300309709e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.1104431037491262e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.1097409600445357e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0940876913188565e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.085512582618426e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.030616795936115e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9371194185482428e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0497240910955826e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0576569384342099e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.8829397194159997e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(Worker pid=6085) INFO:2026-05-20,21:31:51 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 33x across cluster]
(Worker pid=6085) INFO:2026-05-20,21:31:52 ||| Worker at start location (1945,0) completed 50/51 [repeated 2x across cluster]
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.010261393705863e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9976789586685736e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9104384971772155e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.08803110363225e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.1104430922026576e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.072378994209534e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0306171500674292e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9269138823784738e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0467282407739624e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9957130687602292e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.09911847233178e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0306170313022106e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.972642184001478e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.8829397172432835e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.128503515882313e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0880314612602134e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T
(_run_chunk pid=6249) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 1.9104382688687604e-25.
(_run_chunk pid=6249)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T

INFO:2026-05-20,21:32:01 || atm_interpolation.py:atm_interpolation() | Parallel atmospheric interpolations complete.  9.104142189025879 s total, 10984.011225191525 spectra/s, 2746.0028062978813 spectra/s/core

2026-05-20 21:32:01,997	INFO worker.py:1828 -- Calling ray.init() again after it has already been called.

INFO:2026-05-20,21:32:02 || radiative_transfer_engine.py:__init__() | Loading from wavelength_file: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/data/wavelengths.txt

INFO:2026-05-20,21:32:02 || radiative_transfer_engine.py:__init__() | Prebuilt LUT provided

INFO:2026-05-20,21:32:02 || luts.py:load() | Loading LUT into memory

WARNING:2026-05-20,21:32:02 || luts.py:load() | thermal_upwelling is fully NaN, leaving as-is

WARNING:2026-05-20,21:32:02 || luts.py:load() | thermal_downwelling is fully NaN, leaving as-is

INFO:2026-05-20,21:32:02 || radiative_transfer_engine.py:__init__() | LUT grid loaded from file

(_run_chunk pid=6250) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/isofit/utils/atm_interpolation.py:182: LinAlgWarning: An ill-conditioned matrix detected: slice 0 has rcond = 2.0025732372904307e-25. [repeated 1994x across cluster]
(_run_chunk pid=6250)   bhat[i, :] = (inv(X.T @ W @ X) @ X.T @ W @ y).T [repeated 1994x across cluster]
(Worker pid=6517) INFO:2026-05-20,21:32:09 ||| The coszen will be universal:   coszen=0.8430588183136042
(Worker pid=6530) INFO:2026-05-20,21:32:09 ||| The coszen will be universal:   coszen=0.8430588183136042

(Worker pid=6517) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
(Worker pid=6517)   return _methods._mean(a, axis=axis, dtype=dtype,
(Worker pid=6517) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
(Worker pid=6517)   ret = ret.dtype.type(ret / rcount)

(Worker pid=6530) INFO:2026-05-20,21:32:15 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 354x across cluster]

(Worker pid=6531) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice [repeated 3x across cluster]
(Worker pid=6531)   return _methods._mean(a, axis=axis, dtype=dtype, [repeated 3x across cluster]
(Worker pid=6531) /home/runner/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.12/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide [repeated 3x across cluster]
(Worker pid=6531)   ret = ret.dtype.type(ret / rcount) [repeated 3x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:32:20 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:32:25 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 379x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:32:30 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:32:35 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 376x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:32:40 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:32:45 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 379x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:32:50 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 378x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:32:55 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 376x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:00 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 379x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:05 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:10 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:15 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 376x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:20 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 375x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:25 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 376x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:30 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 379x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:35 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:40 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 376x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:45 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 379x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:51 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:33:56 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 376x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:01 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:06 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 378x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:11 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 379x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:16 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:21 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 378x across cluster]

(Worker pid=6531) INFO:2026-05-20,21:34:23 ||| Analytical line writing lines: 51 to 76. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:34:26 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 376x across cluster]

(Worker pid=6532) INFO:2026-05-20,21:34:27 ||| Analytical line writing lines: 25 to 51. Surface: uniform_surface [repeated 2x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:31 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:30 ||| Analytical line writing lines: 76 to 102. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:34:36 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 393x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:41 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:46 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:51 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:34:56 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:01 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:06 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:11 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 378x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:16 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:22 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:27 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:32 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 372x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:37 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:42 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 378x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:47 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 378x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:52 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 371x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:35:57 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 377x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:02 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 377x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:07 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:12 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 377x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:17 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:22 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:27 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 379x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:32 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 372x across cluster]

(Worker pid=6517) INFO:2026-05-20,21:36:37 ||| Analytical line writing lines: 128 to 153. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:36:37 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6532) INFO:2026-05-20,21:36:42 ||| Analytical line writing lines: 153 to 179. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:36:42 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6531) INFO:2026-05-20,21:36:44 ||| Analytical line writing lines: 102 to 128. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:36:47 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:49 ||| Analytical line writing lines: 179 to 205. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:36:52 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:36:57 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 394x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:03 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 394x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:08 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:13 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 396x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:18 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 396x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:23 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:28 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:33 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:38 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:43 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:48 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 397x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:53 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:37:58 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:03 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:08 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:13 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:19 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:24 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:29 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:34 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:39 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:44 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:38:49 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6517) INFO:2026-05-20,21:38:50 ||| Analytical line writing lines: 205 to 230. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:38:54 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6532) INFO:2026-05-20,21:38:54 ||| Analytical line writing lines: 230 to 256. Surface: uniform_surface

(Worker pid=6531) INFO:2026-05-20,21:38:56 ||| Analytical line writing lines: 256 to 282. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:38:59 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:02 ||| Analytical line writing lines: 282 to 307. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:39:04 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:09 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 377x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:14 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:19 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:24 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:29 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:35 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:40 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:45 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:50 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:39:55 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:00 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:05 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:10 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:15 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:20 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:25 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:30 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:35 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:40 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 393x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:45 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:50 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:40:55 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:01 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:06 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6531) INFO:2026-05-20,21:41:06 ||| Analytical line writing lines: 358 to 384. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:41:11 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:11 ||| Analytical line writing lines: 384 to 410. Surface: uniform_surface [repeated 2x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:16 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6517) INFO:2026-05-20,21:41:14 ||| Analytical line writing lines: 307 to 333. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:41:21 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:26 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:31 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:36 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:41 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:46 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:51 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:41:56 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:01 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:06 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:11 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:16 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:22 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:27 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:32 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 393x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:37 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 396x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:42 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:47 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:52 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:42:57 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:02 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:07 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:12 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6532) INFO:2026-05-20,21:43:17 ||| Analytical line writing lines: 435 to 461. Surface: uniform_surface
(Worker pid=6530) INFO:2026-05-20,21:43:17 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:20 ||| Analytical line writing lines: 461 to 487. Surface: uniform_surface
(Worker pid=6530) INFO:2026-05-20,21:43:22 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6531) INFO:2026-05-20,21:43:22 ||| Analytical line writing lines: 410 to 435. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:43:27 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 394x across cluster]

(Worker pid=6517) INFO:2026-05-20,21:43:29 ||| Analytical line writing lines: 487 to 512. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:43:32 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:37 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:43 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 395x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:48 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:53 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:43:58 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:03 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 393x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:08 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:13 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 342x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:18 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 379x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:23 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:28 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:33 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:38 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:43 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:48 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:53 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:44:58 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:03 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:09 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:14 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:19 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:24 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:29 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6532) INFO:2026-05-20,21:45:29 ||| Analytical line writing lines: 512 to 538. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:45:34 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:33 ||| Analytical line writing lines: 538 to 564. Surface: uniform_surface

(Worker pid=6531) INFO:2026-05-20,21:45:37 ||| Analytical line writing lines: 564 to 589. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:45:39 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:44 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:49 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6517) INFO:2026-05-20,21:45:52 ||| Analytical line writing lines: 589 to 615. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:45:54 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:45:59 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:04 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:09 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:14 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:19 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:24 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:30 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:35 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:40 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:45 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:50 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:46:55 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:00 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:05 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:10 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:15 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:20 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:25 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:30 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:35 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6532) INFO:2026-05-20,21:47:37 ||| Analytical line writing lines: 615 to 641. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:47:40 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6531) INFO:2026-05-20,21:47:45 ||| Analytical line writing lines: 666 to 692. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:47:45 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:50 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:47:52 ||| Analytical line writing lines: 641 to 666. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:47:56 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:01 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:06 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6517) INFO:2026-05-20,21:48:10 ||| Analytical line writing lines: 692 to 717. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:48:11 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:16 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:21 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:26 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:31 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:36 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:41 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:46 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:51 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:48:56 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 390x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:01 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:06 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:11 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:16 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:21 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:26 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:31 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:37 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:42 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6532) INFO:2026-05-20,21:49:44 ||| Analytical line writing lines: 717 to 743. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:49:47 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6531) INFO:2026-05-20,21:49:51 ||| Analytical line writing lines: 743 to 769. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:49:52 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:49:57 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:02 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:07 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:11 ||| Analytical line writing lines: 769 to 794. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:50:12 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:17 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:22 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 391x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:27 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:32 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6517) INFO:2026-05-20,21:50:36 ||| Analytical line writing lines: 794 to 820. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:50:37 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 383x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:42 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:47 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:52 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:50:57 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:02 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:07 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:12 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:18 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:23 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:28 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:33 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:38 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:43 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 392x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:48 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6532) INFO:2026-05-20,21:51:51 ||| Analytical line writing lines: 820 to 846. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:51:53 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:51:58 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 376x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:03 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 378x across cluster]

(Worker pid=6531) INFO:2026-05-20,21:52:04 ||| Analytical line writing lines: 846 to 871. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:52:08 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 380x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:13 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 387x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:18 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:23 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 388x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:25 ||| Analytical line writing lines: 871 to 897. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:52:28 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 381x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:33 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:38 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 384x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:44 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 389x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:49 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 386x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:54 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 385x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:52:59 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 382x across cluster]

(Worker pid=6517) INFO:2026-05-20,21:53:00 ||| Analytical line writing lines: 897 to 923. Surface: uniform_surface

(Worker pid=6530) INFO:2026-05-20,21:53:04 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 363x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:09 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 353x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:14 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 354x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:19 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 355x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:24 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 353x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:29 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 352x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:34 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 354x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:39 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 354x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:44 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 353x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:49 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 355x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:53:54 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 356x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:54:00 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 353x across cluster]

(Worker pid=6530) INFO:2026-05-20,21:54:05 ||| Analytical line writing lines: 974 to 1000. Surface: uniform_surface
(Worker pid=6532) INFO:2026-05-20,21:54:05 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 354x across cluster]

INFO:2026-05-20,21:54:08 || analytical_line.py:analytical_line() | Analytical line inversions complete.  1326.28s total, 75.3991 spectra/s, 18.8498 spectra/s/core

INFO:2026-05-20,21:54:08 || apply_oe.py:apply_oe() | Done.

(Worker pid=6532) INFO:2026-05-20,21:54:05 ||| Analytical line writing lines: 923 to 948. Surface: uniform_surface
(Worker pid=6531) INFO:2026-05-20,21:54:08 ||| The coszen will be universal:   coszen=0.8430588183136042 [repeated 125x across cluster]

We can visualize what the segmentation is doing by leveraging the isofit reconstruct_subs CLI command:

EX=$(isofit path examples)/image_cube/medium isofit reconstruct_subs \
    $EX/input/ang20170323t202244_subs_rdn \
    $EX/input/ang20170323t202244_subs_recon_rdn \
    $EX/output/ang20170323t202244_lbl

EX=$(isofit path examples)/image_cube/medium isofit reconstruct_subs \
    $EX/input/ang20170323t202244_subs_loc \
    $EX/input/ang20170323t202244_subs_recon_loc \
    $EX/output/ang20170323t202244_lbl

EX=$(isofit path examples)/image_cube/medium isofit reconstruct_subs \
    $EX/input/ang20170323t202244_subs_obs \
    $EX/input/ang20170323t202244_subs_recon_obs \
    $EX/output/ang20170323t202244_lbl

In [10]:

from isofit.utils.reconstruct import reconstruct_subs

for product in ("rdn", "loc", "obs"):
    reconstruct_subs(
        input_subs_path  = str(env.path("examples", "image_cube", "medium", "input", f"ang20170323t202244_subs_{product}")),
        output_path      = str(env.path("examples", "image_cube", "medium", "input", f"ang20170323t202244_subs_recon_{product}")),
        lbl_working_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_lbl")),
    )

from isofit.utils.reconstruct import reconstruct_subs for product in ("rdn", "loc", "obs"): reconstruct_subs( input_subs_path = str(env.path("examples", "image_cube", "medium", "input", f"ang20170323t202244_subs_{product}")), output_path = str(env.path("examples", "image_cube", "medium", "input", f"ang20170323t202244_subs_recon_{product}")), lbl_working_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_lbl")), )

ERROR:2026-05-20,21:54:09 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_recon_rdn

ERROR:2026-05-20,21:54:10 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_recon_loc

ERROR:2026-05-20,21:54:11 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_recon_obs

Visualizing the input data:

In [11]:

# Plotting the input data
rdn_path = env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_rdn_7k-8k")
loc_path = env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_loc_7k-8k")
obs_path = env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_obs_7k-8k")

subs_rdn_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_rdn")
subs_loc_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_loc")
subs_obs_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_obs")

rdn = envi.open(envi_header(str(rdn_path)))
loc = envi.open(envi_header(str(loc_path)))
obs = envi.open(envi_header(str(obs_path)))

subs_rdn = envi.open(envi_header(str(subs_rdn_path)))
subs_loc = envi.open(envi_header(str(subs_loc_path)))
subs_obs = envi.open(envi_header(str(subs_obs_path)))

rdn_im = rdn.open_memmap(interleave='bip')
loc_im = loc.open_memmap(interleave='bip')
obs_im = obs.open_memmap(interleave='bip')

subs_rdn_im = subs_rdn.open_memmap(interleave='bip')
subs_loc_im = subs_loc.open_memmap(interleave='bip')
subs_obs_im = subs_obs.open_memmap(interleave='bip')

fig, axs = plt.subplots(2, 3, sharex=True, sharey=True, figsize=(10, 10))
axs = np.ravel(axs)

axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15))
axs[1].imshow(loc[0:400, :, 0])
axs[2].imshow(obs[0:400, :, 4])

axs[3].imshow(normalize(subs_rdn_im[0:400, :, bands], 0, 15))
axs[4].imshow(subs_loc[0:400, :, 0])
axs[5].imshow(subs_obs[0:400, :, 4])

axs[0].set_title('Radiance')
axs[1].set_title('Longitude')
axs[2].set_title('Elevation')

axs[0].set_ylabel('Full resolution')
axs[3].set_ylabel('Superpixel resolution')
plt.show()

# Plotting the input data rdn_path = env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_rdn_7k-8k") loc_path = env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_loc_7k-8k") obs_path = env.path("examples", "image_cube", "medium", "data", "ang20170323t202244_obs_7k-8k") subs_rdn_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_rdn") subs_loc_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_loc") subs_obs_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_obs") rdn = envi.open(envi_header(str(rdn_path))) loc = envi.open(envi_header(str(loc_path))) obs = envi.open(envi_header(str(obs_path))) subs_rdn = envi.open(envi_header(str(subs_rdn_path))) subs_loc = envi.open(envi_header(str(subs_loc_path))) subs_obs = envi.open(envi_header(str(subs_obs_path))) rdn_im = rdn.open_memmap(interleave='bip') loc_im = loc.open_memmap(interleave='bip') obs_im = obs.open_memmap(interleave='bip') subs_rdn_im = subs_rdn.open_memmap(interleave='bip') subs_loc_im = subs_loc.open_memmap(interleave='bip') subs_obs_im = subs_obs.open_memmap(interleave='bip') fig, axs = plt.subplots(2, 3, sharex=True, sharey=True, figsize=(10, 10)) axs = np.ravel(axs) axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15)) axs[1].imshow(loc[0:400, :, 0]) axs[2].imshow(obs[0:400, :, 4]) axs[3].imshow(normalize(subs_rdn_im[0:400, :, bands], 0, 15)) axs[4].imshow(subs_loc[0:400, :, 0]) axs[5].imshow(subs_obs[0:400, :, 4]) axs[0].set_title('Radiance') axs[1].set_title('Longitude') axs[2].set_title('Elevation') axs[0].set_ylabel('Full resolution') axs[3].set_ylabel('Superpixel resolution') plt.show()

WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.1097916..1.0983623].

WARNING:2026-05-20,21:54:11 || image.py:_normalize_image_array() | Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.1097916..1.0983623].

We can examine the results from Apply OE after the OE solutions on the superpixels:

In [12]:

wl = np.array(rdn.metadata['wavelength']).astype(float)

state_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_recon_subs_state"))
reconstruct_subs(
    input_subs_path  = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_subs_state")),
    output_path      = state_path,
    lbl_working_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_lbl")),
)

state = envi.open(envi_header(str(state_path)))
state_im = state.open_memmap(interleave='bip')

fig, axs = plt.subplots(1, 4, sharex=True, sharey=True, figsize=(8, 5), tight_layout=True)
axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15))
axs[1].imshow(normalize(state_im[0:400, :, bands], 0, 0.25))
aod = axs[2].imshow(state_im[0:400, :, -2])
h2o = axs[3].imshow(state_im[0:400, :, -1])

plt.colorbar(aod)
plt.colorbar(h2o)

axs[0].set_title('Radiance')
axs[1].set_title('Reflectance')
axs[2].set_title('AOD')
axs[3].set_title('H2O')
plt.show()

wl = np.array(rdn.metadata['wavelength']).astype(float) state_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_recon_subs_state")) reconstruct_subs( input_subs_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_subs_state")), output_path = state_path, lbl_working_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_lbl")), ) state = envi.open(envi_header(str(state_path))) state_im = state.open_memmap(interleave='bip') fig, axs = plt.subplots(1, 4, sharex=True, sharey=True, figsize=(8, 5), tight_layout=True) axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15)) axs[1].imshow(normalize(state_im[0:400, :, bands], 0, 0.25)) aod = axs[2].imshow(state_im[0:400, :, -2]) h2o = axs[3].imshow(state_im[0:400, :, -1]) plt.colorbar(aod) plt.colorbar(h2o) axs[0].set_title('Radiance') axs[1].set_title('Reflectance') axs[2].set_title('AOD') axs[3].set_title('H2O') plt.show()

ERROR:2026-05-20,21:54:11 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/output/ang20170323t202244_recon_subs_state

WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.1097916..1.0983623].

WARNING:2026-05-20,21:54:12 || image.py:_normalize_image_array() | Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.1097916..1.0983623].

WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.12181385..1.5447391].

WARNING:2026-05-20,21:54:12 || image.py:_normalize_image_array() | Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.12181385..1.5447391].

Finally, we can examine the final results at the end of the anlaytical line algorithm:

In [13]:

rfl_path = env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_rfl")
atm_path = env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_atm_interp")

rfl  = envi.open(envi_header(str(rfl_path)))
atm  = envi.open(envi_header(str(atm_path)))

rfl_im = rfl.open_memmap(interleave='bip')
atm_im = atm.open_memmap(interleave='bip')


fig, axs = plt.subplots(1, 4, sharex=True, sharey=True, figsize=(8, 5), tight_layout=True)
axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15))
axs[1].imshow(normalize(rfl_im[0:400, :, bands], 0, 0.25))
aod = axs[2].imshow(atm_im[0:400, :, -2])
h2o = axs[3].imshow(atm_im[0:400, :, -1])

plt.colorbar(aod)
plt.colorbar(h2o)

axs[0].set_title('Radiance')
axs[1].set_title('Reflectance')
axs[2].set_title('AOD')
axs[3].set_title('H2O')
plt.show()

rfl_path = env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_rfl") atm_path = env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_atm_interp") rfl = envi.open(envi_header(str(rfl_path))) atm = envi.open(envi_header(str(atm_path))) rfl_im = rfl.open_memmap(interleave='bip') atm_im = atm.open_memmap(interleave='bip') fig, axs = plt.subplots(1, 4, sharex=True, sharey=True, figsize=(8, 5), tight_layout=True) axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15)) axs[1].imshow(normalize(rfl_im[0:400, :, bands], 0, 0.25)) aod = axs[2].imshow(atm_im[0:400, :, -2]) h2o = axs[3].imshow(atm_im[0:400, :, -1]) plt.colorbar(aod) plt.colorbar(h2o) axs[0].set_title('Radiance') axs[1].set_title('Reflectance') axs[2].set_title('AOD') axs[3].set_title('H2O') plt.show()

WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.1097916..1.0983623].

WARNING:2026-05-20,21:54:13 || image.py:_normalize_image_array() | Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.1097916..1.0983623].

WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.116137974..1.6889473].

WARNING:2026-05-20,21:54:13 || image.py:_normalize_image_array() | Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [0.116137974..1.6889473].