Analysis Software
From Jimenez Group Wiki
If you would like access to the software on this page we ask that you adhere to a few basic conditions:
- You don't share it further than your group, but direct any additional requests to us
- We will answer simple questions and try to add requested features and especially fix bugs, but we can't "support" the software as if it were a commercially available package, we also will not answer basic Igor questions.
- If eventually your group makes improvements, we'd like to have those shared back and ideally keep a master version.
All files on this page require proper credentials for download. Please contact us if you would like access to the files or to report any bugs with the software: password request
Contents
ICARTT
ICARTT Igor Software
| Software | Version | Notes | Links |
|---|---|---|---|
| ICARTT Loader and File Creation Tool | 2023 3.4.2 | K Zarzana, D Sueper and K Aikin |
ICARTT GUI v3 (.ipf) Aug 10 2023 ICARTT Std Name Lookup Tool v1 (.zip) Aug 8 2023
|
ICARTT Igor Software Notes
- These functions create and load data files in the ICARTT format, a widely used, text based format for atmospheric data, particularly for aircraft data. The latest ICARTT specification is linked above. ICARTT is based on the Haines-Hipskind NASA Ames Format, and has undergone several revisions since its inception in 2004. Functions and macros herein are not appropriate for all data generated by ICARTT users; assumptions are outlined below.
- This code can write and read data in a 1D timeseries format, which is the by and large most used ICARTT format ("Fileformat 1001", see below). The ICARTT specification also includes several additional file formats for posting multidimensional data (Fileformats 2110 and 2310), but these are currently not supported by this software.
- There are 2 main versions of the ICARTT specification, v1.1 and v2, the latter being in use since 2017. The specification linked above refers to v2.0. Importantly, the code provided here is able to read and write most standard-compliant v1 and v2 ICARTTs.
- Whenever a user loads or creates a ICARTT format file, an igor data folder called ICARTTFileFormat gets created in the experiment and most of the work is done here. If something goes awry, the user can reset the data folder to root (most everyone's default) by inserting the text below (less the //) into the command line SetDataFolder root:
- Some parameters are calculated; some have simple default values. The later are:
- FileFormatType = 1001 // this value is assumed for reading and creating files.
- FileVolumeNumber = 1 // this value is assumed for reading and creating files.
- TotalVolumeNumber = 1 // this value is assumed for reading and creating files.
- ScaleFactor = 1 // this value is assumed for creating files, but reading files with non-unity scaling factors is supported
- MissingDataIndicator = -9999 // NOTE - This is adjusted depending on the significant digits of the datastream
- None of the original waves in the experiment are modified in any way. Waves selected to be saved are duplicated in the ICARTTFileFormat data folder. The UTC timewave (the independent parameter) and any dependent parameters formatted with a "t" (from the create panel) will have the starting date subtracted from it (in the created file) and will be formatted with one digit after the decimal.
- When loading an icartt file into an experiment, the missing value as reported in the header will be used to convert missing values in the incoming waves to nans. The independent variable (time) will have the UTC date (as indicated in the header) added to it.
- When loading data from the LoadDataOnly menu item, almost no format checking is done and missing data isn't changed to nans.
- The name of the wave in the experiment will be the name given to the dependent and independent parameters, with the exception of matricies, where column names much as mymatrix0, mymatrix1, etc are used.
- If anything is named in the experiment in a 'liberal' fashion (i.e. wave names with spaces), all bets are off.
- Users should be aware that there are operating system differences for end of line designations. Mac uses carriage return (CR), Unix uses line feed (LF), and Windows use carriage return + line feed (CRLF) For icartt, the preferred end of line designation is CRLF (which correspondes to \r\n in the code).
- NASA has been working since 2016 on a general naming specification for atmospheric variables, documented here. An Igor GUI to browse and find suitable standard names for your application is provided in the table above.
Other Igor ICARTT Software
- ICARTT_AMS_FormattingTool_V2_0.ipf. This tool was written by James Allan (U Manachester) and uses one ipf.
- Christoph Knote's Python package for reading and writing ICARTT formatted files – fully documented and tested. On pypi.
ICARTT Matlab Software
The following software was written by Glenn Wolfe (NASA GDFL), who also maintains it, please contact him for support. It provides a generic reader and a specific/example writer procedure:
ICARTT R Software
The following ICARTT reader was written by Steve Wofsy (Harvard), who also maintains it, please contact him for support.
ICARTT Python Software
Provided by Christoph Knote (being updated as of 2022)
- Project page at Pypi Python Package
- Project page at U Augsburg ICARTT Python Package and Discussion
Provided by: Barron Henderson barronh@gmail.com (21-Aug-2013)
- ICARTT reader / netcdf converter for python
- Part of PseudoNetCDF package found here
# In a python script
from PseudoNetCDF.icarttfiles import ffi1001
f = ffi1001('pathtofile')
var = f.variables['HO2_pptv']
print var[:].mean(), var.units
# From command line: dump CDF (like ncdump)
python -m PseudoNetCDF.pncdump -f ffi1001 pathtofile
# From command line: make a NetCDF file (like ncdump file1 | ncgen file2)
python -m PseudoNetCDF.pncgen -f ffi1001 pathtofile
pathtonetcdfoutput.nc
How to read NetCDF v3 into Igor
- Provided by Pedro Campuzano-Jost, with credit to Pengfei Yu.
- This applies to working with instrument/model output that is formatted in netcdf v3 (1997 release).
- Reading netcdf3 files in Igor is extremely painful. There is a 2004, 32-bit compatible XOP from a Japanese group buried in the depths of IgorExchange, with no hope of an update, as well as some code on top of that XOP from both the Manchester guys and Donna. But regardless, it requires you to open a 32-bit version of Igor just to load the file, and the handling of 2D waves and wave attributes is...less than ideal.
- On the other hand, netcdf v4 is very easy to deal with since the data structure conforms to the hdf5 format (used by Aerodyne and Tofwerk instruments and many other applications). But despite the fact that v4 is 10 years old (v 4.0.0 of the netcdf library was released in 2008), many scripts used by atmospheric chemist still use v3 (including about 80% of global modelers). The fix is actually surprisingly easy:
- Download the netcdf 4.6 for windows library/utility pack: http://www.unidata.ucar.edu/downloads/netcdf/ftp/netCDF4.6.1-NC4-64.exe (NCAR, for unknown reasons does not advertise the windows version on their main software page, so if you want to stay up to date check this webpage: https://www.unidata.ucar.edu/software/netcdf/docs/winbin.html)
- Install the package and MAKE SURE you add the library to your System/DOS path (installing for all users is probably the best/easiest option)
- Open a command prompt and cd yourself into the directory where your netcdf v3 file lives (if this sounds scary, just put your file in your personal folder, above "Documents", and you won't need to cd at all)
- Run "nccopy -4 MyNetCDFv3File.nc MyNetCDFv3File.h5"
- The new file you created, MyNetCDFv3File.h5 is now a netcdf4/h5 file and can be perused at will with the hdf5browser in Igor (warning, it is likely quite a bit bigger than the old one).
Mie Code
- The Igor IPF below contains the J-Group Scattering/Extinction calculator, based on the code provided in the book by C. F. Bohren and D. R. Huffman, Absorption and Scattering of Light by Small Particles.
- Our group uses this function frequently to independently check the quantification of the AMS, by e.g. comparing the mass scattering efficiency (MSE) calculated from AMS mass divided by dry PM1 scattering, to the MSE calculated from measured size distributions (e.g. from an SMPS) and this Mie code. For an example see Figure 2d in DeCarlo et al., 2008
- The original version of the Mie subroutine was translated from Fortran into Igor by Charles Brock, NOAA ESRL, and it is provided "as is" (few comments). For interpretation of what that subroutine is doing see the B&H book and also e.g. this link that provided Fortran code with more detailed comments.
- The wrapper code was written by Ed Dunlea, Mike Cubison and Pedro Campuzano Jost.
- Contact Pedro.CampuzanoJost-1@colorado.edu and jose.jimenez@colorado.edu for feedback or bug reports.
- You can download the code from this link.
Picarro G2401
Picarro Software
| Software | Version | Notes | Link |
|---|---|---|---|
| Picarro G2401 Loader and Data Plotting | v1.0 | First release by Michael Lechner | Picarro Panel (.ipf) |
Picarro Software Notes
- Picarro Panel v1.0 (MJL 02/05/2013)
- Recursively loads all .DAT Picarro acquisition files or a specified date range of files in a given directory (best to point the loading directory to DataLog_User and leave the sub directories in their original configuration)
- Plots:
- Picarro measurement time series
- CO is the fully corrected measurement which corrects for CO2 and H2O spectroscopic interferences, as well as H2O dilution.
- CO2_dry is corrected for H2O spectroscopic interference & dilution effect of H2O
- CO2 is not corrected for anything (probably never use)
- CH4_dry is only corrected for H2O spectroscopic interference & dilution effect of H2O
- CH4 is not corrected for anything (probably never use)
- H2O
- Picarro diagnostic time series
- Das Temperature
- Etalon Temperature
- Warm Box Temperature
- Cavity Temperature
- Cavity Pressure
- Picarro measurement time series
Software Output
Measurement Timeseries
Diagnostic Timeseries
Real Time AMS Igor Display
Real Time Software
| Software | Version | Notes | Link |
|---|---|---|---|
| Real Time AMS Igor Display | v1.0 | First release Donna Sueper and Mike Cubison Modified by Michael Lechner |
DAQ_QuickView.ipf |
Real Time Notes
- THIS FUNCTION WILL NOT COMPILE WITHOUT APPROPRIATE XOP!
- An XOP (external operation) is a modular chunk of code that adds features to Igor One needs the file called VDT2.xop or a shortcut to this file to be placed in the directory Wavemetrics:Igor Pro Folder:Igor Extensions: The VDT2.xop is normally found in the folder called Wavemetrics:Igor Pro Folder:More Extensions:Data Acquisition
- Real Time (RT) and Serial Feed (SF) ipf
- This code is for use with the ToF AMS data acquisition program. In the daq one has the option of outputting a limited set of parameters to one text file (TimeTrace.txt)many individual text files (RT_xxxxx.txt) and/or serial output (No file name, just a an old fashioned serial RS232 output. Most will only work with only the RT_xxx.txt files
- The functions in this ipf are intended to run in an Igor experiment concurrently with the DAQ. The RT_xxxx.txt files generated by the DAQ are intended to be read into an Igor experiment on the DAQ computer. Absolutely nothing in this ipf should change anything that happens in the DAQ
- Usage Instructions:
- Aircraft / Ground Plot will bring up a plot of predetermined parameters focused on Aircraft / Ground data
- Delete Real Time Files checkbox when unchecked files are read into igor and then the file is moved to the processed folder. When checked file is not moved to the process folder, but instead is deleted.
- The "Clear.." button will clear existing waves - will set them to zero points. "Pause.." will resume appending values to existing waves.
- The "Process MS itx files" checkbox causes the itx mass spectra files to be loaded and appended to a 2D m/z vs time wave (assuming they have been selected to be saved in the DAQ menu). No plotting or further averaging is done in the DAQ_QuickView ipf posted here.
- The TimeTrace.txt nd the serial feed functions are intended to more specialized purposes.
- The TimeTrace.txt file is intended to be read into an Igor experiment as one file, and no special code is needed (beyond that of creating an igor-friendly time wave). In this case, just create the Real Time panel and press the begin button.
- The serial feed functions are intended to work on a computer that has a serial stream connection to the daq. In this case, just create the Serial Feed panel and press the begin button. In either case, a standard graph of values will be plotted. Users may make new plots or change the existing one to suit their needs.
Software Output
Aircraft Data Plot
SMPS
SMPS Software
The SMPS Software is generally not shared at this point, except with close collaborators of our group, as we don't have time to support it.
| Software | Version | Notes | Link |
|---|---|---|---|
| SMPS data loader/plotter | v4 | First release Doug Day (built on MJC's original loader) | JG_SMPS_v4.pxt Example experiment Sample AIM exported text files |
| SMPS data loader/plotter | v4.46 (latest) |
6 Aug. 2014 -Revised so that only supporting ipf needed is DS_GenTools.ipf (included in pxt posted here). See version notes in header of main .ipf |
JG_SMPS.ipf JG_SMPS_v4_46.pxt |
SMPS Software Notes
JG_SMPS_V4.46 (dd 8/06/14)
- Loads and Concatenates SMPS data collected with TSI AIM (built on Mike C's BEARPEX/CalNex JG_SMPS_v3.ipf and used ToolsMJC_v1a.ipf and Tools_20100221.ipf)
- As of V4.46 uses DS_GenTools.ipf for general functions (previous to then, used JG_GenTools or DougTools.ipf))
- Input should be .txt files created as ouput in TSI AIM software (row-formatted) and dNdLogDp and in named in format: 2010_12_13_1522_SMPS.txt (yyyy_mm_dd_hhmm_SMPS.txt)
- If starting with .ipf (rather than .pxt, .pxp);
- 0) Compile
- 1) Run function "PopulateSMPSpanel()" (which populates the panel with default input values in "root:Temporary")
- 2)"SMPS Plotting Panel" from "SMPS" menu dropdown
- 3) If using a mask wave, a) customize "AssignFlags()" function and run; b) load mask and time waves in "Masterlogger folder"
- "Load-Concatenate" button prompts user to select folderpath for text files to load and also promps user to input assign "project" name to files loaded (will put all data in folder names as such)
- Choose the version of TSI AIM used to export text files (diagnostic columns vary)
- User can chose number of Dp bins to regrid dNdLogDp to at the loading step only. This is useful for when file of different Dp ranges are loaded together. (when plotting later can chose to use these or the native bins). Warning, uses a linear interpolation, not binning, so if much fewer bins than in original data is chosen,not all data points are used in regridding so statistics may be degraded. Thus best to chose similar Dp number to original data and use raw data for final workups, instrument comparison, etc.
- Image Time Series Plots plots either number or volume distributions
- Optional inputs for:
- Dp smoothing
- Max percentile of data assigned to hottest color in image (help use full color range)
- Interpolation across data gaps (of user-specified duration)
- Choosing whether to use regridded or raw dXdLogDp (native scaling)
- Optional inputs for:
- Integrated # and Mass Concentration Plot (Number or Mass)
- Optional inputs: Max/Min Dp over which to integrate total concentration
Additional Notes:
- Best to reselect "Data Folder to Plot" to be sure that datafolder is set to proper project folder (should be reset after all plotting, so only really needed if user changes datafolders). However, must select before first plot.
- Panel inputs stored in root:Temporary (DO NOT DELETE this folder) - can delete all other folders
- Scaled dXdlogDp waves: Currently, these are not used anywhere in the code. To avoid make large "scaled waves" (interpolated to new/fixed timestamp) see the line near the top of the main ipf: "Variable offset = date2secs(2010,12,13), delta=60 //DD". As written (in the posted version) this would make a scaled size distribution from Dec, 13, 2010 with a point every 60 seconds up until YOUR last scan time. To avoid make this long, unused, mostly empty wave, change the date to the day before you collected your data and maybe change the delta to something bigger, like 300+. Then those waves will be small. Will code with more elegant solution in future version.
- There are several auxillary functions at the bottom that may be useful for QC, exporting data. Use "CutBadDat()" function to "manually" enter bad periods that thereafter will be cut when using panel buttons.
- Version update notes at top of .ipf
Next Version Steps:
- Several. See top of ipf. Feel free to send suggestions to Doug.
Software Output
Panel v4.44
dNdlogDp Size Distribution
dMdlogDp Size Distribution
dN Total Concentration
dM Total Concentration
KinSim
KinSim Software
| Software | Version | Notes | Link |
|---|---|---|---|
| KinSim | v2 | First release by Harald Stark 10 Nov. 2014 Modified by Zhe Peng |
KinSim_v2.1.ipf |
| KinSim | v3 | Release by Zhe Peng and Jose-Luis Jimenez 31 May 2018 Modified by Zhe Peng |
KinSim_v3.75.ipf |
KinSim Software Notes
KinSim_v2.1.ipf (ZP 11/10/14)
- Version 1 (by Harald Stark): mechanism generation and integration of rate equations.
- Version 2: functions in v1 and
- P- and T-dependent rate constants
- Advanced input of initial concentrations - compatible to direct input of numbers (for concentrations in molec/cm3), and mixing ratios in "%", "ppth", "ppm", "ppb", "ppt", and "ppq"
- Optimized parameters for the integrator - to handle stiff problems
- Version 2.1: update of help notes and fix of minor bugs
- General usage: compile the procedure then select "Display_help" from the "KinSim" menu. After the dialog box is prompted, select "Basic version" if only the functions in v1 are needed, else select "Version 2". Follow the help notes.
KinSim_v3.75.ipf (ZP 5/31/18)
- Version 3: switch from menu control to user-friedly panel control, substantial function reorganization, and
- Automatic input check (for basic parameters and mechanism)
- Species concentration constrain
- Special treatment for photolysis, emissions, and dilutions (user-defined dilution air composition)
- Times series plots of species concentrations and reaction rates
- Version 3.75: fix of bugs, conversion of some macros to functions, and
- Mechanism export and import
- Preclusion of species from dilution
- Mechanism input in both JPL and IUPAC formats
- Possibility of enabling/disabling reactions in the mechanism
- Fractional product stoichiometric coefficients
- Option of simple rate-coefficient input table
- Translation table to build time-dependence table
- Real-time T & P change
- Gas-particle partitioning calculations
- General usage: see here
Input Mechanisms
- Basic OFR mechanism (v1.00 ZP 9/10/14)
- Including photolytic, Ox, HOx, NOx, and SO2 reactions in OFRs
- A few reactions have special rate constant expressions. Refer to the "Notes" column for more details.
- Link: Basic_OFR_mechanism.pxp
- Mechanism of CH4 full oxidation in OFRs (v1.00 ZP 9/10/14)
- Should be used along with the basic OFR mechanism
- Link: CH4_OFR-oxidation_mechanism.pxp
- Mechanism for 2014 NO3-Terpene Chamber Studies (v1.00 DD, 6 Jan, 2015)
- Link: NO3_Terpene_mechanism.pxp
- Should work with latest KinSim versions, however see here for ipfs pre-version control, including additional post-processing/plotting functions and example plots images.
- Mechanism for IEPOX-SOA (v1.00 DJ, 17 Nov, 2017)
- IEPOX-SOA mechanism based on Marais et al. (2016)
- Link: IEPOX_SOA_mechanism.pxp
Other Utilities
- List/post other utilities here
