Analyzer interface
cibmangotree.analyzer_interface
¶
Modules:
| Name | Description |
|---|---|
column_automap |
|
context |
|
data_type_compatibility |
|
declaration |
|
interface |
|
params |
|
Attributes¶
Classes¶
Functions:¶
Modules¶
column_automap
¶
Functions:
| Name | Description |
|---|---|
check_name_hint |
Returns true if every word in the hint (split by spaces) is present in the name, |
column_automap |
Matches user-provided columns to the expected columns based on the name hints. |
Attributes¶
Classes¶
Functions:¶
check_name_hint(name, hint)
¶
Returns true if every word in the hint (split by spaces) is present in the name, in a case insensitive manner.
Source code in src/cibmangotree/analyzer_interface/column_automap.py
52 53 54 55 56 57 | |
column_automap(user_columns, input_schema_columns)
¶
Matches user-provided columns to the expected columns based on the name hints.
The resulting dictionary is keyed by the expected input column name.
Source code in src/cibmangotree/analyzer_interface/column_automap.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
context
¶
Classes:
| Name | Description |
|---|---|
AssetsReader |
|
BaseDerivedModuleContext |
Common interface for secondary analyzers runtime contexts. |
InputTableReader |
|
NullProgressReporter |
No-op progress reporter for when progress reporting is disabled. |
PrimaryAnalyzerContext |
|
ProgressReporterProtocol |
Protocol for progress reporting during analysis steps. |
SecondaryAnalyzerContext |
|
TableReader |
|
TableWriter |
|
Classes¶
AssetsReader
¶
Bases: ABC
Methods:
| Name | Description |
|---|---|
table |
Gets the table reader for the specified output. |
Source code in src/cibmangotree/analyzer_interface/context.py
128 129 130 131 132 133 134 | |
table(output_id)
abstractmethod
¶Gets the table reader for the specified output.
Source code in src/cibmangotree/analyzer_interface/context.py
129 130 131 132 133 134 | |
BaseDerivedModuleContext
¶
Bases: ABC, BaseModel
Common interface for secondary analyzers runtime contexts.
Methods:
| Name | Description |
|---|---|
dependency |
Gets the context of a secondary analyzer the current module depends on, which |
Attributes:
| Name | Type | Description |
|---|---|---|
base |
AssetsReader
|
Gets the base primary analyzer's context, which lets you inspect and load its |
base_params |
dict[str, ParamValue]
|
Gets the primary analysis parameters. |
temp_dir |
str
|
Gets the temporary directory that the module can freely write content to |
Source code in src/cibmangotree/analyzer_interface/context.py
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 | |
base
abstractmethod
property
¶Gets the base primary analyzer's context, which lets you inspect and load its outputs.
base_params
abstractmethod
property
¶Gets the primary analysis parameters.
temp_dir
instance-attribute
¶Gets the temporary directory that the module can freely write content to during its lifetime. This directory will not persist between runs.
dependency(secondary_interface)
abstractmethod
¶Gets the context of a secondary analyzer the current module depends on, which lets you inspect and load its outputs.
Source code in src/cibmangotree/analyzer_interface/context.py
108 109 110 111 112 113 114 115 116 | |
InputTableReader
¶
Bases: TableReader
Methods:
| Name | Description |
|---|---|
preprocess |
Given the manually loaded user input dataframe, apply column mapping and |
Source code in src/cibmangotree/analyzer_interface/context.py
151 152 153 154 155 156 157 158 159 160 161 | |
preprocess(df)
abstractmethod
¶Given the manually loaded user input dataframe, apply column mapping and semantic transformations to give the input dataframe that the analyzer expects.
Source code in src/cibmangotree/analyzer_interface/context.py
152 153 154 155 156 157 158 159 160 161 | |
NullProgressReporter
¶
No-op progress reporter for when progress reporting is disabled.
Source code in src/cibmangotree/analyzer_interface/context.py
28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | |
PrimaryAnalyzerContext
¶
Bases: ABC, BaseModel
Methods:
| Name | Description |
|---|---|
input |
Gets the input reader context. |
output |
Gets the output writer context for the specified output ID. |
Attributes:
| Name | Type | Description |
|---|---|---|
params |
dict[str, ParamValue]
|
Gets the analysis parameters. |
temp_dir |
str
|
Gets the temporary directory that the module can freely write content to |
Source code in src/cibmangotree/analyzer_interface/context.py
47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | |
params
abstractmethod
property
¶Gets the analysis parameters.
temp_dir
instance-attribute
¶Gets the temporary directory that the module can freely write content to during its lifetime. This directory will not persist between runs.
input()
abstractmethod
¶Gets the input reader context.
Note that this is in function form even though one input is expected, in anticipation that we may want to support multiple inputs in the future.
Source code in src/cibmangotree/analyzer_interface/context.py
54 55 56 57 58 59 60 61 62 | |
output(output_id)
abstractmethod
¶Gets the output writer context for the specified output ID.
Source code in src/cibmangotree/analyzer_interface/context.py
72 73 74 75 76 77 | |
ProgressReporterProtocol
¶
Bases: Protocol
Protocol for progress reporting during analysis steps.
Methods:
| Name | Description |
|---|---|
finish |
Mark the step as complete. |
update |
Update progress value (0.0 to 1.0). |
Source code in src/cibmangotree/analyzer_interface/context.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 | |
finish(done_text='Done!')
¶Mark the step as complete.
Source code in src/cibmangotree/analyzer_interface/context.py
19 20 21 | |
update(value)
¶Update progress value (0.0 to 1.0).
Source code in src/cibmangotree/analyzer_interface/context.py
15 16 17 | |
SecondaryAnalyzerContext
¶
Bases: BaseDerivedModuleContext
Methods:
| Name | Description |
|---|---|
output |
Gets the output writer context |
Source code in src/cibmangotree/analyzer_interface/context.py
119 120 121 122 123 124 125 | |
TableReader
¶
Bases: ABC
Attributes:
| Name | Type | Description |
|---|---|---|
parquet_path |
str
|
Gets the path to the table's parquet file. The module should expect a parquet |
Source code in src/cibmangotree/analyzer_interface/context.py
137 138 139 140 141 142 143 144 145 | |
TableWriter
¶
Bases: ABC
Attributes:
| Name | Type | Description |
|---|---|---|
parquet_path |
str
|
Gets the path to the table's parquet file. The module should write a parquet |
Source code in src/cibmangotree/analyzer_interface/context.py
164 165 166 167 168 169 170 171 172 | |
data_type_compatibility
¶
Functions:
| Name | Description |
|---|---|
get_data_type_compatibility_score |
Returns a score for the compatibility of the actual data type with the |
Attributes:
| Name | Type | Description |
|---|---|---|
data_type_mapping_preference |
dict[DataType, list[list[DataType]]]
|
For each data type, a list of lists of data types that are considered compatible |
Attributes¶
data_type_mapping_preference = {'text': [['text'], ['identifier', 'url']], 'integer': [['integer']], 'float': [['float', 'integer']], 'boolean': [['boolean']], 'datetime': [['datetime']], 'time': [['time'], ['datetime']], 'identifier': [['identifier'], ['url', 'datetime'], ['integer'], ['text']], 'url': [['url']]}
module-attribute
¶
For each data type, a list of lists of data types that are considered compatible with it. The first list is the most preferred, the last list is the least. The items in each list are considered equally compatible.
Functions:¶
get_data_type_compatibility_score(expected_data_type, actual_data_type)
¶
Returns a score for the compatibility of the actual data type with the
expected data type. Higher (less negative) scores are better.
None means the data types are not compatible.
Source code in src/cibmangotree/analyzer_interface/data_type_compatibility.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | |
declaration
¶
Classes:
| Name | Description |
|---|---|
AnalyzerDeclaration |
|
SecondaryAnalyzerDeclaration |
|
Classes¶
AnalyzerDeclaration
¶
Bases: AnalyzerInterface
Methods:
| Name | Description |
|---|---|
__init__ |
Creates a primary analyzer declaration |
Source code in src/cibmangotree/analyzer_interface/declaration.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
__init__(interface, main, *, is_distributed=False, default_params=lambda _: dict())
¶Creates a primary analyzer declaration
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
interface
¶ |
AnalyzerInterface
|
The metadata interface for the primary analyzer. |
required |
main
¶ |
Callable
|
The entry point function for the primary analyzer. This function should
take a single argument of type |
required |
is_distributed
¶ |
bool
|
Set this explicitly to |
False
|
Source code in src/cibmangotree/analyzer_interface/declaration.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
SecondaryAnalyzerDeclaration
¶
Bases: SecondaryAnalyzerInterface
Methods:
| Name | Description |
|---|---|
__init__ |
Creates a secondary analyzer declaration |
Source code in src/cibmangotree/analyzer_interface/declaration.py
52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | |
__init__(interface, main)
¶Creates a secondary analyzer declaration
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
interface
¶ |
SecondaryAnalyzerInterface
|
The metadata interface for the secondary analyzer. |
required |
main
¶ |
Callable
|
The entry point function for the secondary analyzer. This function should
take a single argument of type |
required |
Source code in src/cibmangotree/analyzer_interface/declaration.py
55 56 57 58 59 60 61 62 63 64 65 66 | |
interface
¶
Classes:
| Name | Description |
|---|---|
AnalyzerInterface |
|
AnalyzerOutput |
|
AnalyzerParam |
|
BaseAnalyzerInterface |
|
DerivedAnalyzerInterface |
|
InputColumn |
|
SecondaryAnalyzerInterface |
|
Attributes:
| Name | Type | Description |
|---|---|---|
DataType |
The semantic data type for a data column. This is not quite the same as |
Attributes¶
DataType = Literal['text', 'integer', 'float', 'boolean', 'datetime', 'identifier', 'url', 'time']
module-attribute
¶
The semantic data type for a data column. This is not quite the same as structural data types like polars or pandas or even arrow types, but they represent how the data is intended to be interpreted.
textis expected to be a free-form human-readable text content.integerandfloatare meant to be manipulated arithmetically.booleanis a binary value.datetimerepresents time and are meant to be manipulated as time values.timerepresents time within a day, not including the date information.identifieris a unique identifier for a record. It is not expected to be manipulated in any way.urlis a string that represents a URL.
Classes¶
AnalyzerInterface
¶
Bases: BaseAnalyzerInterface
Attributes:
| Name | Type | Description |
|---|---|---|
input |
AnalyzerInput
|
Specifies the input data schema for the analyzer. |
outputs |
list[AnalyzerOutput]
|
Specifies the output data schema for the analyzer. |
params |
list[AnalyzerParam]
|
A list of parameters that the analyzer accepts. |
Source code in src/cibmangotree/analyzer_interface/interface.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 | |
AnalyzerOutput
¶
Bases: BaseModel
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
Uniquely identifies the output data schema for the analyzer. The analyzer |
name |
str
|
The human-friendly for the output. |
Source code in src/cibmangotree/analyzer_interface/interface.py
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | |
AnalyzerParam
¶
Bases: BaseModel
Attributes:
| Name | Type | Description |
|---|---|---|
backfill_value |
Optional[ParamValue]
|
Recommended if this is a parameter that is newly introduced in a previously |
default |
Optional[ParamValue]
|
Optional: define a static default value for this parameter. A parameter |
description |
Optional[str]
|
A short description of the parameter. This is used in the UI to represent |
human_readable_name |
Optional[str]
|
The human-friendly name for the parameter. This is used in the UI to |
id |
str
|
The name of the parameter. This becomes the key in the parameters dictionary |
type |
ParamType
|
The type of the parameter. This is used for validation and for customizing |
Source code in src/cibmangotree/analyzer_interface/interface.py
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | |
backfill_value = None
class-attribute
instance-attribute
¶Recommended if this is a parameter that is newly introduced in a previously released analyzer. The backfill is show what this parameter was before it became customizable.
default = None
class-attribute
instance-attribute
¶Optional: define a static default value for this parameter. A parameter without a default will need to be chosen explicitly by the user.
description = None
class-attribute
instance-attribute
¶A short description of the parameter. This is used in the UI to represent the parameter.
human_readable_name = None
class-attribute
instance-attribute
¶The human-friendly name for the parameter. This is used in the UI to represent the parameter.
id
instance-attribute
¶The name of the parameter. This becomes the key in the parameters dictionary that is passed to the analyzer.
type
instance-attribute
¶The type of the parameter. This is used for validation and for customizing the UX for parameter input.
BaseAnalyzerInterface
¶
Bases: BaseModel
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
The static ID for the analyzer that, with the version, uniquely identifies the |
long_description |
Optional[str]
|
A longer description of what the analyzer does that will be shown separately. |
name |
str
|
The short human-readable name of the analyzer. |
short_description |
str
|
A short, one-liner description of what the analyzer does. |
version |
str
|
The version ID for the analyzer. In future, we may choose to support output |
Source code in src/cibmangotree/analyzer_interface/interface.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | |
id
instance-attribute
¶The static ID for the analyzer that, with the version, uniquely identifies the analyzer and will be stored as metadata as part of the output data.
long_description = None
class-attribute
instance-attribute
¶A longer description of what the analyzer does that will be shown separately.
name
instance-attribute
¶The short human-readable name of the analyzer.
short_description
instance-attribute
¶A short, one-liner description of what the analyzer does.
version
instance-attribute
¶The version ID for the analyzer. In future, we may choose to support output migration between versions of the same analyzer.
DerivedAnalyzerInterface
¶
Bases: BaseAnalyzerInterface
Attributes:
| Name | Type | Description |
|---|---|---|
base_analyzer |
AnalyzerInterface
|
The base analyzer that this secondary analyzer extends. This is always a primary |
depends_on |
list[SecondaryAnalyzerInterface]
|
A dictionary of secondary analyzers that must be run before the current analyzer |
Source code in src/cibmangotree/analyzer_interface/interface.py
141 142 143 144 145 146 147 148 149 150 151 152 153 154 | |
base_analyzer
instance-attribute
¶The base analyzer that this secondary analyzer extends. This is always a primary
analyzer. If your module depends on other secondary analyzers (which must have
the same base analyzer), you can specify them in the depends_on field.
depends_on = []
class-attribute
instance-attribute
¶A dictionary of secondary analyzers that must be run before the current analyzer secondary analyzer is run. These secondary analyzers must have the same primary base.
InputColumn
¶
Bases: Column
Attributes:
| Name | Type | Description |
|---|---|---|
name_hints |
list[str]
|
Specifies a list of space-separated words that are likely to be found in the |
Source code in src/cibmangotree/analyzer_interface/interface.py
194 195 196 197 198 199 200 201 202 203 | |
name_hints = []
class-attribute
instance-attribute
¶Specifies a list of space-separated words that are likely to be found in the column name of the user-provided data. This is used to help the user map the input columns to the expected columns.
Any individual hint matching is sufficient for a match to be called. The hint in turn is matched if every word matches some part of the column name.
SecondaryAnalyzerInterface
¶
Bases: DerivedAnalyzerInterface
Attributes:
| Name | Type | Description |
|---|---|---|
outputs |
list[AnalyzerOutput]
|
Specifies the output data schema for the analyzer. |
Source code in src/cibmangotree/analyzer_interface/interface.py
157 158 159 160 161 162 163 | |
params
¶
Classes:
| Name | Description |
|---|---|
IntegerParam |
Represents an integer value |
TimeBinningParam |
Represents a time bin. |
TimeBinningValue |
|
Classes¶
IntegerParam
¶
Bases: BaseModel
Represents an integer value
The corresponding value will be of type int.
Source code in src/cibmangotree/analyzer_interface/params.py
16 17 18 19 20 21 22 23 24 25 | |
TimeBinningParam
¶
Bases: BaseModel
Represents a time bin.
The corresponding value will be of type TimeBinningValue.
Source code in src/cibmangotree/analyzer_interface/params.py
28 29 30 31 32 33 34 35 | |
TimeBinningValue
¶
Bases: BaseModel
Methods:
| Name | Description |
|---|---|
to_polars_truncate_spec |
Converts the value to a string that can be used in Polars truncate spec. |
Source code in src/cibmangotree/analyzer_interface/params.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | |
to_polars_truncate_spec()
¶Converts the value to a string that can be used in Polars truncate spec. See https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.truncate.html
Source code in src/cibmangotree/analyzer_interface/params.py
44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | |