Apply OE¶
Apply OE is the built-in end-to-end atmospheric correction pipeline, and is the easiest way to run Isofit. Apply OE is run via a command line interface (CLI) tool accessed with a terminal call:
isofit apply_oe --help
The function docstring is printed when this command is run within a terminal window, and acts as a guide for how to run the function.
from isofit.utils.apply_oe import apply_oe
print(apply_oe.__doc__)
Applies OE over a flightline using a radiative transfer engine. This executes
ISOFIT in a generalized way, accounting for the types of variation that might be
considered typical.
Observation (obs) and location (loc) files are used to determine appropriate
geometry lookup tables and provide a heuristic means of determining atmospheric
water ranges.
Parameters
----------
input_radiance : str
Radiance data cube. Expected to be ENVI format
input_loc : str
Location data cube of shape (Lon, Lat, Elevation). Expected to be ENVI format
input_obs : str
Observation data cube of shape:
(path length, to-sensor azimuth, to-sensor zenith,
to-sun azimuth, to-sun zenith, phase,
slope, aspect, cosine i, UTC time)
Expected to be ENVI format
working_directory : str
Directory to stage multiple outputs, will contain subdirectories
sensor : str
The sensor used for acquisition, will be used to set noise and datetime
settings
surface_path : str
Path to surface model or json dict of surface model configuration
copy_input_files : bool, default=False
Flag to choose to copy input_radiance, input_loc, and input_obs locally into
the working_directory
modtran_path : str, default=None
Location of MODTRAN utility. Alternately set with `MODTRAN_DIR` environment
variable
wavelength_path : str, default=None
Location to get wavelength information from, if not specified the radiance
header will be used
surface_category : str, default="multicomponent_surface"
The type of ISOFIT surface priors to use. Default is multicomponent_surface
aerosol_climatology_path : str, default=None
Specific aerosol climatology information to use in MODTRAN
rdn_factors_path : str, default=None
Specify a radiometric correction factor, if desired
atmosphere_type : str, default="ATM_MIDLAT_SUMMER"
Atmospheric profile to be used for MODTRAN simulations. Unused for other
radiative transfer models.
channelized_uncertainty_path : str, default=None
Path to a wavelength-specific channelized uncertainty file - used to augment Sy in the OE formalism
instrument_noise_path : str, default=None
Path to a wavelength-specific instrument noise file, used to derive per-wavelength NEDL / SNR
dn_uncertainty_file: str, default=None
Path to a linearity .mat file to augment S matrix with linearity uncertainty
model_discrepancy_path : str, default=None
Modifies S_eps in the OE formalism as the Gamma additive term, as:
S_eps = Sy + Kb.dot(self.Sb).dot(Kb.T) + Gamma
lut_config_file : str, default=None
Path to a look up table configuration file, which will override defaults
choices
multiple_restarts : bool, default=False
Use multiple initial starting poitns for each OE ptimization run, using
the corners of the atmospheric variables as starting points. This gives
a more robust, albeit more expensive, solution.
logging_level : str, default="INFO"
Logging level with which to run ISOFIT
log_file : str, default=None
File path to write ISOFIT logs to
n_cores : int, default=1
Number of cores to run ISOFIT with. Substantial parallelism is available, and
full runs will be very slow in serial. Suggested to max this out on the
available system
presolve : int, default=False
Flag to use a presolve mode to estimate the available atmospheric water range.
Runs a preliminary inversion over the image with a 1-D LUT of water vapor, and
uses the resulting range (slightly expanded) to bound determine the full LUT.
Advisable to only use with small cubes or in concert with the empirical_line
setting, or a significant speed penalty will be incurred
empirical_line : bool, default=False
Use an empirical line interpolation to run full inversions over only a subset
of pixels, determined using a SLIC superpixel segmentation, and use a KDTREE of
local solutions to interpolate radiance->reflectance. Generally a good option
if not trying to analyze the atmospheric state at fine scale resolution.
Mutually exclusive with analytical_line
analytical_line : bool, default=False
Use an analytical solution to the fixed atmospheric state to solve for each
pixel. Starts by running a full OE retrieval on each SLIC superpixel, then
interpolates the atmospheric state to each pixel, and closes with the
analytical solution.
Mutually exclusive with empirical_line
ray_temp_dir : str, default="/tmp/ray"
Location of temporary directory for ray parallelization engine
emulator_base : str, default=None
Location of emulator base path. Point this at the 3C or 6C .h5 files.
sRTMnet to use the emulator instead of MODTRAN. An additional file with the
same basename and the extention _aux.npz must accompany
e.g. /path/to/emulator.h5 /path/to/emulator_aux.npz
segmentation_size : int, default=40
If empirical_line is enabled, sets the size of segments to construct
num_neighbors : list[int], default=[]
Forced number of neighbors for empirical line extrapolation - overides default
set from segmentation_size parameter
atm_sigma : list[int], default=[2]
A list of smoothing factors to use during the atmospheric interpolation, one
for each atmospheric parameter (or broadcast to all if only one is provided).
Only used with the analytical line.
pressure_elevation : bool, default=False
Flag to retrieve elevation
prebuilt_lut : str, default=None
Use this pre-constructed look up table for all retrievals. Must be an
ISOFIT-compatible RTE NetCDF
no_min_lut_spacing : bool, default=False
Don't allow the LUTConfig to remove a LUT dimension because of minimal spacing.
inversion_windows : list[float], default=None
Override the default inversion windows. Will supercede any sensor specific
defaults that are in place.
Must be in 2-item tuples
config_only : bool, default=False
Generates the configuration then exits before execution. If presolve is
enabled, that run will still occur.
interpolate_bad_rdn : bool, default=False
Flag to perform a per-pixel interpolation across no-data and NaN data bands.
Does not interpolate vectors that are entire no-data or NaN, only partial.
Currently only designed for wavelength interpolation on spectra.
Does NOT do any spatial interpolation
interpolate_inplace : bool, default=False
Flag to tell interpolation to work on the file in place, or generate a
new interpolated rdn file. The location of the new file will be in the
"input" directory within the working directory.
skyview_factor : str, default=None
Flag to determine method to account for skyview factor. Default is None, creating an array of 1s.
Other option is "slope" which will approx. based on cos^2(slope/2).
Other option is a path to a skyview ENVI file computed via skyview.py utility or other source.
Please note data must range from 0-1.
resources : bool, default=False
Enables the system resource tracker. Must also have the log_file set.
retrieve_co2 : bool, default=False
Flag to retrieve CO2 in the state vector. Only available with emulator at the moment.
eof_path : str, default=None
Add 1 or 2 Empirical Orthogonal Functions to the state vector. File is a 1-2 column text file
with one number per instrument channel.
terrain_style : str, default=dem
Flag to set the terrain style. dem uses provided obs values, flat sets the surface to the spheroid
References
----------
D.R. Thompson, A. Braverman,P.G. Brodrick, A. Candela, N. Carbon, R.N. Clark,D. Connelly, R.O. Green, R.F.
Kokaly, L. Li, N. Mahowald, R.L. Miller, G.S. Okin, T.H.Painter, G.A. Swayze, M. Turmon, J. Susilouto, and
D.S. Wettergreen. Quantifying Uncertainty for Remote Spectroscopy of Surface Composition. Remote Sensing of
Environment, 2020. doi: https://doi.org/10.1016/j.rse.2020.111898.
sRTMnet emulator:
P.G. Brodrick, D.R. Thompson, J.E. Fahlen, M.L. Eastwood, C.M. Sarture, S.R. Lundeen, W. Olson-Duvall,
N. Carmon, and R.O. Green. Generalized radiative transfer emulation for imaging spectroscopy reflectance
retrievals. Remote Sensing of Environment, 261:112476, 2021.doi: 10.1016/j.rse.2021.112476.
The Apply OE function can leverage a large number of input parameters, but most are optional. The important inputs are the non-optional arguments:
INPUT_RADIANCE INPUT_LOC INPUT_OBS WORKING_DIRECTORY SENSOR and --surface_path.
which must be entered in the specified order (except --surface_path which can be added at any point but still required). Descriptions of each are found in the docstring printed above. It is important to note that the INPUT_RADIANCE, INPUT_LOC, and INPUT_OBS are ENVI raster data formats that must be at the same row-column dimensions. The --surface_path points Isofit torwards the constructed prior distribution file for surface reflectance. Optional arguments are denoted by the '--' in their name, e.g. --modtran_path, --pressure_elevation. It is important to note that the default radiative transfer engine (RTE) is currently set to Modtran. You must specify an --emulator_path without a Modtran installation.
How do you run Apply OE?¶
The script is run via the CLI. For example:
IC=$(isofit path examples)/image_cube/smallsets the environment variableICto the output of the commandisofit path examples/image_cube/small. This is just used to reuse the same output for multiple arguments. Depending on the user's isofit installation environment, this path may vary. This command ensures the correct path is retrieved
IC=$(isofit path examples)/image_cube/small EX=$(isofit path examples)/image_cube/small \
isofit apply_oe \
$IC/ang20170323t202244_rdn_7000-7010 \
$IC/ang20170323t202244_loc_7000-7010 \
$IC/ang20170323t202244_obs_7000-7010 \
$EX \
ang \
--surface_path $EX/configs/surface.json \
--emulator_base $(isofit path srtmnet --key file) \
--n_cores 10 \
--presolve \
Here,
$IC/ang20170323t202244_rdn_7000-7010
$IC/ang20170323t202244_loc_7000-7010
$IC/ang20170323t202244_obs_7000-7010
are the radiance, location, and observational geometry files respectively. The \ tells the CLI call to expect a multi-line input. The remaining two requried parameters are ang, the sensor designation (AVIRIS-NG), and the --surface_path pointing to the surface configuration file which will be built at runtime.
The remaining arguments set Apply OE to run with:
--emulator_basepoints isofit to the location of the sRTMnet emulator to usse as the radiative transfer engine (RTE)--n_cores = 10CPU cores- The
--presolvealgorithm to narrow down the water vapor retrievals
# Alternatively, you can programatically call apply_oe. If you executed apply_oe via the command line above, do not run this
# We use the env object to retrieve the actual location of the ISOFIT extra dependencies instead of assuming where they could be
import os
import shutil
from isofit.utils.apply_oe import apply_oe
from isofit.data import env
output = env.path("examples", "image_cube", "small")
# Cleanup any previous runs; comment this out if you want to preserve a previous run's output
if (o := output / "output").exists():
shutil.rmtree(o)
apply_oe(
input_radiance = str(env.path("examples", "image_cube", "small", "ang20170323t202244_rdn_7000-7010")),
input_loc = str(env.path("examples", "image_cube", "small", "ang20170323t202244_loc_7000-7010")),
input_obs = str(env.path("examples", "image_cube", "small", "ang20170323t202244_obs_7000-7010")),
working_directory = str(output),
sensor = "ang",
surface_path = str(env.path("examples", "image_cube", "small", "configs", "surface.json")),
emulator_base = str(env.path("srtmnet", key="srtmnet.file")),
presolve = True,
n_cores = os.cpu_count(),
)
The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/ang20170323t202244_rdn_7000-7010
The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/ang20170323t202244_loc_7000-7010
The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/ang20170323t202244_obs_7000-7010
INFO:2026-02-24,21:02:13 || apply_oe.py:apply_oe() | Checking input data files...
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[2], line 15 12 if (o := output / "output").exists(): 13 shutil.rmtree(o) ---> 15 apply_oe( 16 input_radiance = str(env.path("examples", "image_cube", "small", "ang20170323t202244_rdn_7000-7010")), 17 input_loc = str(env.path("examples", "image_cube", "small", "ang20170323t202244_loc_7000-7010")), 18 input_obs = str(env.path("examples", "image_cube", "small", "ang20170323t202244_obs_7000-7010")), 19 working_directory = str(output), 20 sensor = "ang", 21 surface_path = str(env.path("examples", "image_cube", "small", "configs", "surface.json")), 22 emulator_base = str(env.path("srtmnet", key="srtmnet.file")), 23 presolve = True, 24 n_cores = os.cpu_count(), 25 ) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/isofit/utils/apply_oe.py:330, in apply_oe(input_radiance, input_loc, input_obs, working_directory, sensor, surface_path, copy_input_files, modtran_path, wavelength_path, surface_category, surface_class_file, classify_multisurface, aerosol_climatology_path, rdn_factors_path, atmosphere_type, channelized_uncertainty_path, instrument_noise_path, dn_uncertainty_file, model_discrepancy_path, lut_config_file, multiple_restarts, logging_level, log_file, n_cores, presolve, empirical_line, analytical_line, ray_temp_dir, emulator_base, segmentation_size, num_neighbors, atm_sigma, pressure_elevation, prebuilt_lut, no_min_lut_spacing, inversion_windows, config_only, interpolate_bad_rdn, interpolate_inplace, skyview_factor, resources, retrieve_co2, eof_path, terrain_style) 325 logging.error( 326 "The resources.jsonl will only be generated when a log file is also set" 327 ) 329 logging.info("Checking input data files...") --> 330 rdn_dataset = envi.open(envi_header(input_radiance)) 331 rdn_size = (rdn_dataset.shape[0], rdn_dataset.shape[1]) 332 del rdn_dataset File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/envi.py:304, in open(file, image) 272 def open(file, image=None): 273 ''' 274 Opens an image or spectral library with an associated ENVI HDR header file. 275 (...) 301 Capitalized versions of the file extensions are also searched. 302 ''' --> 304 header_path = find_file_path(file) 305 h = read_envi_header(header_path) 306 check_compatibility(h) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/spyfile.py:122, in find_file_path(filename) 118 if not pathname: 119 msg = 'Unable to locate file "%s". If the file exists, ' \ 120 'use its full path or place its directory in the ' \ 121 'SPECTRAL_DATA environment variable.' % filename --> 122 raise FileNotFoundError(msg) 123 return pathname FileNotFoundError: Unable to locate file "/home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/ang20170323t202244_rdn_7000-7010.hdr". If the file exists, use its full path or place its directory in the SPECTRAL_DATA environment variable.
We can examine both the inputs and outputs of Apply OE with this run call:
# Common imports
import os
from pathlib import Path
from spectral import envi
from matplotlib import pyplot as plt
import numpy as np
from isofit.core.common import envi_header
from isofit.data import env
INFO:2026-02-24,21:02:14 || font_manager.py:__init__() | Failed to extract font properties from /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf: Can not load face (unknown file format; error code 0x2)
INFO:2026-02-24,21:02:14 || font_manager.py:_load_fontmanager() | generated new fontManager
# Load the input files
rdn_path = env.path("examples", "image_cube", "small", "ang20170323t202244_rdn_7000-7010")
loc_path = env.path("examples", "image_cube", "small", "ang20170323t202244_loc_7000-7010")
obs_path = env.path("examples", "image_cube", "small", "ang20170323t202244_obs_7000-7010")
rdn = envi.open(envi_header(str(rdn_path)))
loc = envi.open(envi_header(str(loc_path)))
obs = envi.open(envi_header(str(obs_path)))
rdn_im = rdn.open_memmap(interleave='bip')
loc_im = loc.open_memmap(interleave='bip')
obs_im = obs.open_memmap(interleave='bip')
ERROR:2026-02-24,21:02:14 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/ang20170323t202244_rdn_7000-7010
ERROR:2026-02-24,21:02:14 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/ang20170323t202244_loc_7000-7010
ERROR:2026-02-24,21:02:14 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/ang20170323t202244_obs_7000-7010
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[4], line 6 3 loc_path = env.path("examples", "image_cube", "small", "ang20170323t202244_loc_7000-7010") 4 obs_path = env.path("examples", "image_cube", "small", "ang20170323t202244_obs_7000-7010") ----> 6 rdn = envi.open(envi_header(str(rdn_path))) 7 loc = envi.open(envi_header(str(loc_path))) 8 obs = envi.open(envi_header(str(obs_path))) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/envi.py:304, in open(file, image) 272 def open(file, image=None): 273 ''' 274 Opens an image or spectral library with an associated ENVI HDR header file. 275 (...) 301 Capitalized versions of the file extensions are also searched. 302 ''' --> 304 header_path = find_file_path(file) 305 h = read_envi_header(header_path) 306 check_compatibility(h) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/spyfile.py:122, in find_file_path(filename) 118 if not pathname: 119 msg = 'Unable to locate file "%s". If the file exists, ' \ 120 'use its full path or place its directory in the ' \ 121 'SPECTRAL_DATA environment variable.' % filename --> 122 raise FileNotFoundError(msg) 123 return pathname FileNotFoundError: Unable to locate file "/home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/ang20170323t202244_rdn_7000-7010.hdr". If the file exists, use its full path or place its directory in the SPECTRAL_DATA environment variable.
# Print the bands of the input files
print('Band names in the location file:')
[print(f"{i}") for i in loc.metadata['band names']]
print()
print('Band names in the observational geometry file:')
temp = [print(f"{i}") for i in obs.metadata['band names']]
Band names in the location file:
--------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[5], line 3 1 # Print the bands of the input files 2 print('Band names in the location file:') ----> 3 [print(f"{i}") for i in loc.metadata['band names']] 5 print() 6 print('Band names in the observational geometry file:') NameError: name 'loc' is not defined
# Plot the input data
normalize = lambda x, vmin, vmax: (x - vmin) / (vmax - vmin)
bands = [55, 35, 15]
fig, axs = plt.subplots(1, 3, sharex=True, sharey=True, figsize=(13, 4))
plot = axs[0].imshow(normalize(rdn_im[..., bands], 0, 15))
plot = axs[1].imshow(loc_im[..., 0])
plot = axs[2].imshow(obs_im[..., 4])
title = axs[0].set_title('Radiance (RGB)')
title = axs[1].set_title('Longitude (WGS-84)')
title = axs[2].set_title('Solar zenith angle (Deg)')
--------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[6], line 6 3 bands = [55, 35, 15] 5 fig, axs = plt.subplots(1, 3, sharex=True, sharey=True, figsize=(13, 4)) ----> 6 plot = axs[0].imshow(normalize(rdn_im[..., bands], 0, 15)) 7 plot = axs[1].imshow(loc_im[..., 0]) 8 plot = axs[2].imshow(obs_im[..., 4]) NameError: name 'rdn_im' is not defined
The input image doesn't look like much because this is just a 10x10 pixel example. However we see per-pixel spectral variation in the radiance RGBs, and systematic variation in the location and geometric variables.
# Load the output files
rfl_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_rfl")
state_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_state")
uncert_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_uncert")
rfl = envi.open(envi_header(str(rfl_path)))
state = envi.open(envi_header(str(state_path)))
uncert = envi.open(envi_header(str(uncert_path)))
rfl_im = rfl.open_memmap(interleave='bip')
state_im = state.open_memmap(interleave='bip')
uncert_im = uncert.open_memmap(interleave='bip')
print(f'Shape of the _rfl file: {rfl_im.shape}')
print(f'Shape of the _state file: {state_im.shape}')
print(f'Shape of the _uncert file: {uncert_im.shape}')
ERROR:2026-02-24,21:02:14 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/output/ang20170323t202244_rfl
ERROR:2026-02-24,21:02:14 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/output/ang20170323t202244_state
ERROR:2026-02-24,21:02:14 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/output/ang20170323t202244_uncert
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[7], line 6 3 state_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_state") 4 uncert_path = env.path("examples", "image_cube", "small", "output", "ang20170323t202244_uncert") ----> 6 rfl = envi.open(envi_header(str(rfl_path))) 7 state = envi.open(envi_header(str(state_path))) 8 uncert = envi.open(envi_header(str(uncert_path))) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/envi.py:304, in open(file, image) 272 def open(file, image=None): 273 ''' 274 Opens an image or spectral library with an associated ENVI HDR header file. 275 (...) 301 Capitalized versions of the file extensions are also searched. 302 ''' --> 304 header_path = find_file_path(file) 305 h = read_envi_header(header_path) 306 check_compatibility(h) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/spyfile.py:122, in find_file_path(filename) 118 if not pathname: 119 msg = 'Unable to locate file "%s". If the file exists, ' \ 120 'use its full path or place its directory in the ' \ 121 'SPECTRAL_DATA environment variable.' % filename --> 122 raise FileNotFoundError(msg) 123 return pathname FileNotFoundError: Unable to locate file "/home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/small/output/ang20170323t202244_rfl.hdr". If the file exists, use its full path or place its directory in the SPECTRAL_DATA environment variable.
The difference between the _rfl file and the _state file is that the _rfl file only contains the solutions for surface reflectance variables. Here, the AVIRIS-NG image contains 425 wavelength bands. As a result, the _rfl contains 425 bands. The _state and _uncert files contain the surface reflectance solutions and uncertainty calculated as the standard deviation of the posterior distributions for the 425 wavelength bands and for non-reflectance statevector elements; here, aerosol optical depth (AOD) and water vapor (H2O).
# Plot the output data
normalize = lambda x, vmin, vmax: (x - vmin) / (vmax - vmin)
bands = [55, 35, 15]
fig, axs = plt.subplots(2, 2, sharex=True, sharey=True, figsize=(8, 8))
axs = np.ravel(axs)
plot = axs[0].imshow(normalize(rfl_im[..., bands], 0, 0.3))
uncert = axs[1].imshow(uncert_im[..., 55])
aod = axs[2].imshow(state_im[..., -2])
h2o = axs[3].imshow(state_im[..., -1])
plt.colorbar(uncert)
plt.colorbar(aod)
plt.colorbar(h2o)
title = axs[0].set_title('Reflectance')
title = axs[1].set_title('Uncertainty (at 650 nm)')
title = axs[2].set_title('AOD')
title = axs[3].set_title('H2O')
--------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[8], line 7 5 fig, axs = plt.subplots(2, 2, sharex=True, sharey=True, figsize=(8, 8)) 6 axs = np.ravel(axs) ----> 7 plot = axs[0].imshow(normalize(rfl_im[..., bands], 0, 0.3)) 8 uncert = axs[1].imshow(uncert_im[..., 55]) 9 aod = axs[2].imshow(state_im[..., -2]) NameError: name 'rfl_im' is not defined
The Analytical Line¶
In the example above, we ran Isofit using the full optimal estimation (OE) on each pixel independently. Computationally, this amounts to iterating through each row-column pair to solve for the full state-vector (427 variables in the above case). However for large images, this demands long run-times, and ignores the principle that some state-vector elements, namely the atmospheric variables like AOD and H2O, should not vary from one pixel to another, but rather should be spatially smooth, and only vary over multi-pixel length scales Link to the relevent paper.
The analytical line and empirical line algorithms leverage the assumption of a spatially smooth atmosphere to decrease run times by a factor of 10. Currently, we suggest using the analytical line algorithm and not the empirical line algorithm.
See the following CLI call to run Apply OE with the analytical line algorithm:
IC=$(isofit path examples)/image_cube/medium EX=$(isofit path examples)/image_cube/medium \
isofit apply_oe \
$IC/ang20170323t202244_rdn_7k-8k \
$IC/ang20170323t202244_loc_7k-8k \
$IC/ang20170323t202244_obs_7k-8k \
$EX \
ang \
--surface_path $EX/configs/surface.json \
--emulator_base $(isofit path srtmnet --key file) \
--n_cores 10 \
--presolve \
--analytical_line \
--segmentation_size 50 \
--log_file $EX/log.txt
Most of the input parameters are identical to the per-pixel application above. However, we've added a --log_file, the --analytical_line flag, and a --segmentation_size. The --log_file points the program to write a text file to print logging statements during run time. The --analytical_line flag tells Isofit to use the analytical line algorithm.
A simple overview for the anylitical line algorithm:
All three input files are "segmented" into superpixel blocks using the SLIC algorithm. The
--segmentation_sizevalue sets the number of pixels that each superpixel contains.At the superpixel resolution, Isofit solves for the OE solutions, which provides both surface and atmospheric state variables.
Atmospheric state variables are spatially interpolated to full image resolution. The spatial interpolation uses the Apply OE parameters
--num_neighborsand--atm_sigma.With a fixed atmosphere, we leverage a closed form solution for surface state elements that allows for a solution convergence in a single iteration.
# Alternatively, you can programatically call apply_oe. If you executed apply_oe via the command line above, do not run this
# We use the env object to retrieve the actual location of the ISOFIT extra dependencies instead of assuming where they could be
import os
import shutil
from isofit.utils.apply_oe import apply_oe
from isofit.data import env
output = env.path("examples", "image_cube", "medium")
# Cleanup any previous runs; comment this out if you want to preserve a previous run's output
if (o := output / "output").exists():
shutil.rmtree(o)
apply_oe(
input_radiance = str(env.path("examples", "image_cube", "medium", "ang20170323t202244_rdn_7000-7010")),
input_loc = str(env.path("examples", "image_cube", "medium", "ang20170323t202244_loc_7000-7010")),
input_obs = str(env.path("examples", "image_cube", "medium", "ang20170323t202244_obs_7000-7010")),
working_directory = str(output),
sensor = "ang",
surface_path = str(env.path("examples", "image_cube", "medium", "configs", "surface.json")),
emulator_base = str(env.path("srtmnet", key="srtmnet.file")),
presolve = True,
n_cores = os.cpu_count(),
analytical_line = True,
segmentation_size = 50,
)
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/ang20170323t202244_rdn_7000-7010
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/ang20170323t202244_loc_7000-7010
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/ang20170323t202244_obs_7000-7010
INFO:2026-02-24,21:02:15 || apply_oe.py:apply_oe() | Checking input data files...
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[9], line 15 12 if (o := output / "output").exists(): 13 shutil.rmtree(o) ---> 15 apply_oe( 16 input_radiance = str(env.path("examples", "image_cube", "medium", "ang20170323t202244_rdn_7000-7010")), 17 input_loc = str(env.path("examples", "image_cube", "medium", "ang20170323t202244_loc_7000-7010")), 18 input_obs = str(env.path("examples", "image_cube", "medium", "ang20170323t202244_obs_7000-7010")), 19 working_directory = str(output), 20 sensor = "ang", 21 surface_path = str(env.path("examples", "image_cube", "medium", "configs", "surface.json")), 22 emulator_base = str(env.path("srtmnet", key="srtmnet.file")), 23 presolve = True, 24 n_cores = os.cpu_count(), 25 analytical_line = True, 26 segmentation_size = 50, 27 ) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/isofit/utils/apply_oe.py:330, in apply_oe(input_radiance, input_loc, input_obs, working_directory, sensor, surface_path, copy_input_files, modtran_path, wavelength_path, surface_category, surface_class_file, classify_multisurface, aerosol_climatology_path, rdn_factors_path, atmosphere_type, channelized_uncertainty_path, instrument_noise_path, dn_uncertainty_file, model_discrepancy_path, lut_config_file, multiple_restarts, logging_level, log_file, n_cores, presolve, empirical_line, analytical_line, ray_temp_dir, emulator_base, segmentation_size, num_neighbors, atm_sigma, pressure_elevation, prebuilt_lut, no_min_lut_spacing, inversion_windows, config_only, interpolate_bad_rdn, interpolate_inplace, skyview_factor, resources, retrieve_co2, eof_path, terrain_style) 325 logging.error( 326 "The resources.jsonl will only be generated when a log file is also set" 327 ) 329 logging.info("Checking input data files...") --> 330 rdn_dataset = envi.open(envi_header(input_radiance)) 331 rdn_size = (rdn_dataset.shape[0], rdn_dataset.shape[1]) 332 del rdn_dataset File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/envi.py:304, in open(file, image) 272 def open(file, image=None): 273 ''' 274 Opens an image or spectral library with an associated ENVI HDR header file. 275 (...) 301 Capitalized versions of the file extensions are also searched. 302 ''' --> 304 header_path = find_file_path(file) 305 h = read_envi_header(header_path) 306 check_compatibility(h) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/spyfile.py:122, in find_file_path(filename) 118 if not pathname: 119 msg = 'Unable to locate file "%s". If the file exists, ' \ 120 'use its full path or place its directory in the ' \ 121 'SPECTRAL_DATA environment variable.' % filename --> 122 raise FileNotFoundError(msg) 123 return pathname FileNotFoundError: Unable to locate file "/home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/ang20170323t202244_rdn_7000-7010.hdr". If the file exists, use its full path or place its directory in the SPECTRAL_DATA environment variable.
We can visualize what the segmentation is doing by leveraging the isofit reconstruct_subs CLI command:
EX=$(isofit path examples)/image_cube/medium isofit reconstruct_subs \
$EX/input/ang20170323t202244_subs_rdn \
$EX/input/ang20170323t202244_subs_recon_rdn \
$EX/output/ang20170323t202244_lbl
EX=$(isofit path examples)/image_cube/medium isofit reconstruct_subs \
$EX/input/ang20170323t202244_subs_loc \
$EX/input/ang20170323t202244_subs_recon_loc \
$EX/output/ang20170323t202244_lbl
EX=$(isofit path examples)/image_cube/medium isofit reconstruct_subs \
$EX/input/ang20170323t202244_subs_obs \
$EX/input/ang20170323t202244_subs_recon_obs \
$EX/output/ang20170323t202244_lbl
from isofit.utils.reconstruct import reconstruct_subs
for product in ("rdn", "loc", "obs"):
reconstruct_subs(
input_subs_path = str(env.path("examples", "image_cube", "medium", "input", f"ang20170323t202244_subs_{product}")),
output_path = str(env.path("examples", "image_cube", "medium", "input", f"ang20170323t202244_subs_recon_{product}")),
lbl_working_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_lbl")),
)
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_rdn
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_recon_rdn
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/output/ang20170323t202244_lbl
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[10], line 4 1 from isofit.utils.reconstruct import reconstruct_subs 3 for product in ("rdn", "loc", "obs"): ----> 4 reconstruct_subs( 5 input_subs_path = str(env.path("examples", "image_cube", "medium", "input", f"ang20170323t202244_subs_{product}")), 6 output_path = str(env.path("examples", "image_cube", "medium", "input", f"ang20170323t202244_subs_recon_{product}")), 7 lbl_working_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_lbl")), 8 ) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/isofit/utils/reconstruct.py:44, in reconstruct_subs(input_subs_path, output_path, lbl_working_path) 27 """Helper function to take the flat array that the superpixel 28 algorithms work with and turn them into images at the full resolution 29 of the input/output file. They will have the full array-resolution, (...) 41 None 42 """ 43 # Load the input data ---> 44 subs_input = envi.open(envi_header(input_subs_path)) 45 subs_input_ar = np.squeeze(subs_input.open_memmap(interleave="bip")) 47 lbl = envi.open(envi_header(lbl_working_path)) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/envi.py:304, in open(file, image) 272 def open(file, image=None): 273 ''' 274 Opens an image or spectral library with an associated ENVI HDR header file. 275 (...) 301 Capitalized versions of the file extensions are also searched. 302 ''' --> 304 header_path = find_file_path(file) 305 h = read_envi_header(header_path) 306 check_compatibility(h) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/spyfile.py:122, in find_file_path(filename) 118 if not pathname: 119 msg = 'Unable to locate file "%s". If the file exists, ' \ 120 'use its full path or place its directory in the ' \ 121 'SPECTRAL_DATA environment variable.' % filename --> 122 raise FileNotFoundError(msg) 123 return pathname FileNotFoundError: Unable to locate file "/home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_rdn.hdr". If the file exists, use its full path or place its directory in the SPECTRAL_DATA environment variable.
Visualizing the input data:
# Plotting the input data
rdn_path = env.path("examples", "image_cube", "medium", "ang20170323t202244_rdn_7000-7010")
loc_path = env.path("examples", "image_cube", "medium", "ang20170323t202244_loc_7000-7010")
obs_path = env.path("examples", "image_cube", "medium", "ang20170323t202244_obs_7000-7010")
subs_rdn_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_rdn")
subs_loc_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_loc")
subs_obs_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_obs")
rdn = envi.open(envi_header(str(rdn_path)))
loc = envi.open(envi_header(str(loc_path)))
obs = envi.open(envi_header(str(obs_path)))
subs_rdn = envi.open(envi_header(str(subs_rdn_path)))
subs_loc = envi.open(envi_header(str(subs_loc_path)))
subs_obs = envi.open(envi_header(str(subs_obs_path)))
rdn_im = rdn.open_memmap(interleave='bip')
loc_im = loc.open_memmap(interleave='bip')
obs_im = obs.open_memmap(interleave='bip')
subs_rdn_im = subs_rdn.open_memmap(interleave='bip')
subs_loc_im = subs_loc.open_memmap(interleave='bip')
subs_obs_im = subs_obs.open_memmap(interleave='bip')
fig, axs = plt.subplots(2, 3, sharex=True, sharey=True, figsize=(10, 10))
axs = np.ravel(axs)
axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15))
axs[1].imshow(loc[0:400, :, 0])
axs[2].imshow(obs[0:400, :, 4])
axs[3].imshow(normalize(subs_rdn_im[0:400, :, bands], 0, 15))
axs[4].imshow(subs_loc[0:400, :, 0])
axs[5].imshow(subs_obs[0:400, :, 4])
axs[0].set_title('Radiance')
axs[1].set_title('Longitude')
axs[2].set_title('Elevation')
axs[0].set_ylabel('Full resolution')
axs[3].set_ylabel('Superpixel resolution')
plt.show()
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/ang20170323t202244_rdn_7000-7010
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/ang20170323t202244_loc_7000-7010
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/ang20170323t202244_obs_7000-7010
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_recon_rdn
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_recon_loc
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/input/ang20170323t202244_subs_recon_obs
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[11], line 10 7 subs_loc_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_loc") 8 subs_obs_path = env.path("examples", "image_cube", "medium", "input", "ang20170323t202244_subs_recon_obs") ---> 10 rdn = envi.open(envi_header(str(rdn_path))) 11 loc = envi.open(envi_header(str(loc_path))) 12 obs = envi.open(envi_header(str(obs_path))) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/envi.py:304, in open(file, image) 272 def open(file, image=None): 273 ''' 274 Opens an image or spectral library with an associated ENVI HDR header file. 275 (...) 301 Capitalized versions of the file extensions are also searched. 302 ''' --> 304 header_path = find_file_path(file) 305 h = read_envi_header(header_path) 306 check_compatibility(h) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/spyfile.py:122, in find_file_path(filename) 118 if not pathname: 119 msg = 'Unable to locate file "%s". If the file exists, ' \ 120 'use its full path or place its directory in the ' \ 121 'SPECTRAL_DATA environment variable.' % filename --> 122 raise FileNotFoundError(msg) 123 return pathname FileNotFoundError: Unable to locate file "/home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/ang20170323t202244_rdn_7000-7010.hdr". If the file exists, use its full path or place its directory in the SPECTRAL_DATA environment variable.
We can examine the results from Apply OE after the OE solutions on the superpixels:
wl = np.array(rdn.metadata['wavelength']).astype(float)
state_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_recon_subs_state"))
reconstruct_subs(
input_subs_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_subs_state")),
output_path = state_path,
lbl_working_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_lbl")),
)
state = envi.open(envi_header(str(state_path)))
state_im = state.open_memmap(interleave='bip')
fig, axs = plt.subplots(1, 4, sharex=True, sharey=True, figsize=(8, 5), tight_layout=True)
axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15))
axs[1].imshow(normalize(state_im[0:400, :, bands], 0, 0.25))
aod = axs[2].imshow(state_im[0:400, :, -2])
h2o = axs[3].imshow(state_im[0:400, :, -1])
plt.colorbar(aod)
plt.colorbar(h2o)
axs[0].set_title('Radiance')
axs[1].set_title('Reflectance')
axs[2].set_title('AOD')
axs[3].set_title('H2O')
plt.show()
--------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[12], line 1 ----> 1 wl = np.array(rdn.metadata['wavelength']).astype(float) 3 state_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_recon_subs_state")) 4 reconstruct_subs( 5 input_subs_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_subs_state")), 6 output_path = state_path, 7 lbl_working_path = str(env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_lbl")), 8 ) NameError: name 'rdn' is not defined
Finally, we can examine the final results at the end of the anlaytical line algorithm:
rfl_path = env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_rfl")
atm_path = env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_atm_interp")
rfl = envi.open(envi_header(str(rfl_path)))
atm = envi.open(envi_header(str(atm_path)))
rfl_im = rfl.open_memmap(interleave='bip')
atm_im = atm.open_memmap(interleave='bip')
fig, axs = plt.subplots(1, 4, sharex=True, sharey=True, figsize=(8, 5), tight_layout=True)
axs[0].imshow(normalize(rdn_im[0:400, :, bands], 0, 15))
axs[1].imshow(normalize(rfl_im[0:400, :, bands], 0, 0.25))
aod = axs[2].imshow(atm_im[0:400, :, -2])
h2o = axs[3].imshow(atm_im[0:400, :, -1])
plt.colorbar(aod)
plt.colorbar(h2o)
axs[0].set_title('Radiance')
axs[1].set_title('Reflectance')
axs[2].set_title('AOD')
axs[3].set_title('H2O')
plt.show()
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/output/ang20170323t202244_rfl
ERROR:2026-02-24,21:02:15 || ini.py:path() | The following path does not exist, please verify your installation environment: /home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/output/ang20170323t202244_atm_interp
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[13], line 4 1 rfl_path = env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_rfl") 2 atm_path = env.path("examples", "image_cube", "medium", "output", "ang20170323t202244_atm_interp") ----> 4 rfl = envi.open(envi_header(str(rfl_path))) 5 atm = envi.open(envi_header(str(atm_path))) 7 rfl_im = rfl.open_memmap(interleave='bip') File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/envi.py:304, in open(file, image) 272 def open(file, image=None): 273 ''' 274 Opens an image or spectral library with an associated ENVI HDR header file. 275 (...) 301 Capitalized versions of the file extensions are also searched. 302 ''' --> 304 header_path = find_file_path(file) 305 h = read_envi_header(header_path) 306 check_compatibility(h) File ~/work/isofit-tutorials/isofit-tutorials/.venv/lib/python3.13/site-packages/spectral/io/spyfile.py:122, in find_file_path(filename) 118 if not pathname: 119 msg = 'Unable to locate file "%s". If the file exists, ' \ 120 'use its full path or place its directory in the ' \ 121 'SPECTRAL_DATA environment variable.' % filename --> 122 raise FileNotFoundError(msg) 123 return pathname FileNotFoundError: Unable to locate file "/home/runner/work/isofit-tutorials/isofit-tutorials/image_cube/medium/output/ang20170323t202244_rfl.hdr". If the file exists, use its full path or place its directory in the SPECTRAL_DATA environment variable.