datasets

Dataset loading module

Part of scilightcon.

Classes

`scilightcon.datasets.LogsReader`

Reader object for getting time-dependent data from logs folders, created by different software (Argos, CEP, ThermoLoggers, etc.)

Examples:

>>> from scilightcon.datasets import LogsReader
>>> import datetime
>>> directory = r'\\konversija\kleja\ThermologgerLogs\v5'
>>> reader = LogsReader(directory)
>>> loggers_names_list = reader.list_loggers()
>>> loggers_names_list
['Location 2B 314', 'Location 2D 3.14 Logger 1-4', 'Location 2D 3.14 Logger 5-8', ...]
>>> logger_name = 'Location 2B 314'
>>> measurables_list = reader.list_measurables(logger_name)
>>> measurables_list
['A1-H Stalas 1', 'A1-H', 'A1-T Stalas 1', 'A1-T']
>>> measurable = 'A1-H Stalas 1'
>>> from_date = datetime.datetime(2023,7,20)
>>> to_date = datetime.datetime(2023,7,21)
>>> times, values = reader.get_data(logger_name=logger_name, measurable=measurable, from_date=from_date, to_date=to_date)

Methods:

`scilightcon.datasets.LogsReader.get_data(logger_name, measurable, from_date=None, to_date=None)`

Function checks if given logger_name and measurable are valid and collects timestamps and values for a given time period.

Parameters:

Name	Type	Description	Default
`logger_name`	`str`	Logger name, for example: "Location 2B 314"	required
`measurable`	`str`	Measurable name, for example: "A1-H Stalas 1"	required
`from_date`	`datetime`	Date from which the data will be collected	`None`
`to_date`	`datetime`	Date to which the data will be collected	`None`

Returns:

Name	Type	Description
`times`	`List(datetime)`	A list with timestamps
`values`	`List(float)`	A list with the values

`scilightcon.datasets.LogsReader.list_loggers()`

Collects names of available loggers

Returns:

Type	Description
`List[str]`	A list of Logger names

`scilightcon.datasets.LogsReader.list_measurables(logger_name)`

Collects names of measurables of a given logger

Parameters:

Name	Type	Description	Default
`logger_name`	`str`	Logger name	required

Returns:

Type	Description
`List[str]`	A list of measurables that can be found for the specific logger

Functions:

`scilightcon.datasets.load_atmospheric_data()`

Loads atmospheric data.

Examples:

>>> from scilightcon.datasets import load_atmospheric_data
>>> data, header = load_atmospheric_data()

Returns:

Name	Type	Description
`data`	`Ndarray`	A 2D array of data with headers excluded. Shape (n_samples, n_columns)
`header`	`List`	Column names or empty strings. Shape (n_columns)

`scilightcon.datasets.load_chroma_spectrum(filter, mode='t')`

Load a Chroma catalog optic transmission or reflection spectrum from the scilightcon database.

Examples:

>>> from scilightcon.datasets import load_chroma_spectrum
>>> data, header = load_chroma_spectrum('ET340X', 't')
>>> np.shape(data)
(1601, 2)
>>> header
['Wavelength  (nm)', 'Transmission (%)']

Parameters:

Name	Type	Description	Default
`filter`	`str`	`ET340X`	required
`mode`	`str`	`t`, `r`	`'t'`

Returns:

Name	Type	Description
`data`	`Ndarray`	A 2D array of data with headers excluded. Shape (n_samples, n_columns)
`header`	`List`	Column names or empty strings. Shape (n_columns)

`scilightcon.datasets.load_csv_data(data_file_name, *, data_module=DATA_MODULE)`

Loads data_file_name from data_module with importlib.resources.

Examples:

>>> from scilightcon.datasets import load_csv_data
>>> data, header = load_csv_data('Hg_lines.csv')

Parameters:

Name	Type	Description	Default
`data_file_name`	`str`	Name of csv file to be loaded from `data_module/data_file_name`.	required
`data_module`	`str or module`	Module where data lives. The default is `'scilightcon.datasets.data'`	`DATA_MODULE`

Returns:

Name	Type	Description
`data`	`ndarray`	A 2D array with each row representing one sample and each column representing the features of a given sample. Shape: n_samples, n_features
`target`	`ndarry`	A 1D array holding target variables for all the samples in `data`. For example target[0] is the target variable for data[0]. Shape (n_samples,)
`target_names`	`ndarry`	A 1D array containing the names of the classifications. For example target_names[0] is the name of the target[0] class. Shape (n_samples,)

`scilightcon.datasets.load_edmund_spectrum(filter, mode)`

Load an Edmund catalog optic transmission or reflection spectrum from the scilightcon database.

Examples:

>>> from scilightcon.datasets import load_edmund_spectrum
>>> data, header = load_edmund_spectrum('69202', 't')
>>> np.shape(data)
(346, 2)
>>> header
['Wavelength (nm)', 'Transmission (%)']

Parameters:

Name	Type	Description	Default
`filter`	`str`	`lp_400nm`, `lp_450nm`, `lp_500nm`, `lp_550nm`, `lp_600nm`, `lp_600nm`, `lp_700nm`, `lp_750nm`, `sp_400nm`, `sp_500nm`, `sp_600nm` or `sp_700nm`	required

Returns:

Name	Type	Description
`data`	`Ndarray`	A 2D array of data with headers excluded. Shape (n_samples, n_columns)
`header`	`List`	Column names or empty strings. Shape (n_columns)

`scilightcon.datasets.load_eksma_spectrum(material)`

Load an Eksma catalog optic transmission or reflection spectrum from the scilightcon database.

Examples:

>>> from scilightcon.datasets import load_eksma_spectrum
>>> data, header = load_eksma_spectrum('Ag')
>>> np.shape(data)
(172, 2)
>>> header
['Wavelength (nm)', 'Reflection (%)']

Parameters:

Name	Type	Description	Default
`material`	`str`	`Ag`, `Au` or `Al`	required

Returns:

Name	Type	Description
`data`	`Ndarray`	A 2D array of data with headers excluded. Shape (n_samples, n_columns)
`header`	`List`	Column names or empty strings. Shape (n_columns)

`scilightcon.datasets.load_hamamatsu_spectrum(detector)`

Load a Hamamatsu detector quantum efficiency spectrum from the scilightcon database.

Examples:

>>> from scilightcon.datasets import load_hamamatsu_spectrum
>>> data, header = load_hamamatsu_spectrum('H10721-210')
>>> np.shape(data)
(1200, 2)
>>> header
['Wavelength  (nm)', 'QE (%)']

Parameters:

Name	Type	Description	Default
`detector`	`str`	`H7422-40`, `H7422-50`, `H10721-01`, `H10721-110`	required

Returns:

Name	Type	Description
`data`	`Ndarray`	A 2D array of data with headers excluded. Shape (n_samples, n_columns)
`header`	`List`	Column names or empty strings. Shape (n_columns)

`scilightcon.datasets.load_spectrum(vendor, name, mode)`

Load a transmission, reflection or quantum efficiency spectrum for a named optic or detector from one of the vendor datasets bundled with scilightcon.

This is the generic loader behind load_hamamatsu_spectrum, load_thorlabs_spectrum, load_eksma_spectrum and load_chroma_spectrum. Prefer those vendor-specific functions when the vendor is known ahead of time: their name argument typed via Literal to available filters and detectors.

Examples:

>>> from scilightcon.datasets import load_spectrum
>>> data, header = load_spectrum('thorlabs', 'DMLP425', mode='t')
>>> np.shape(data)
(2251, 2)
>>> header
['Wavelength  (nm)', 'Transmission (%)']

Parameters:

Name	Type	Description	Default
`source`	`str`	`hamamatsu`, `thorlabs`, `eksma` or `chroma`.	required
`name`	`str`	Vendor-specific filter/detector/material name, e.g. `DMLP425` for `thorlabs`. See `load_hamamatsu_spectrum`, `load_thorlabs_spectrum`, `load_eksma_spectrum` and `load_chroma_spectrum` for the valid names per vendor.	required
`mode`	`str`	`t` for transmission, `r` for reflection. Ignored by vendors that only publish one direction (`hamamatsu` QE curves, `eksma` mirror reflections).	required

Returns:

Name	Type	Description
`data`	`Ndarray`	A 2D array of data with headers excluded. Shape (n_samples, n_columns)
`header`	`List`	Column names or empty strings. Shape (n_columns)

`scilightcon.datasets.load_thorlabs_spectrum(filter, mode='t')`

Load a Thorlabs catalog optic transmission or reflection spectrum from the scilightcon database.

Examples:

>>> from scilightcon.datasets import load_thorlabs_spectrum
>>> data, header = load_thorlabs_spectrum('DMLP425')
>>> np.shape(data)
(2251, 2)
>>> header
['Wavelength  (nm)', 'Transmission (%)']

Parameters:

Name	Type	Description	Default
`filter`	`str`	`DMLP425`, `DMLP550`, `DMLP650`, `FB340-10`, `FBH343-10`,	required
`mode`	`str`	`t`, `r`	`'t'`

Returns:

Name	Type	Description
`data`	`Ndarray`	A 2D array of data with headers excluded. Shape (n_samples, n_columns)
`header`	`List`	Column names or empty strings. Shape (n_columns)

`scilightcon.datasets.load_zipped_csv_data(data_file_name, *, data_module=DATA_MODULE)`

Extracts gzip file to csv.

Examples:

>>> from scilightcon.datasets import load_zipped_csv_data
>>> data_file_name = r'C:\Code\lightcon-scipack\scilightcon\datasets\data\data_test_detect_peaks.csv.gz'
>>> data, header = _load_zipped_csv_data(data_file_name)

Parameters:

Name	Type	Description	Default
`data_file_name`	`str`	Path of the file that needs to be extracted	required
`data_module`	`str or module`	Module where data lives. The default is `'scilightcon.datasets.data'`	`DATA_MODULE`

Returns:

Name	Type	Description
`data`	`Ndarray`	A 2D array of data with headers excluded. Shape (n_samples, n_columns)
`header`	`List`	Column names or empty strings. Shape (n_columns)