Skip to content

cibmangotree.analyzers

Modules:

Name Description
hashtags
ngrams
time_coordination

Modules

hashtags

Modules:

Name Description
hashtags_base
Modules
hashtags_base

Modules:

Name Description
main
Classes Modules
main

Functions:

Name Description
gini

Parameters

Classes Functions:
gini(x)
Parameters

x : pl.Series polars Series containing values for which to compute the Gini coefficient

Returns

float Gini coefficient (between 0.0 and 1.0)

Source code in src/cibmangotree/analyzers/hashtags/hashtags_base/main.py
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
def gini(x: pl.Series) -> float:
    """
    Parameters
    ----------
    x : pl.Series
        polars Series containing values for which to compute the Gini coefficient

    Returns
    -------
    float
        Gini coefficient (between 0.0 and 1.0)
    """
    sorted_x = x.value_counts().sort(by="count", descending=False)[:, 1].to_list()

    n = len(sorted_x)
    cumx = list(accumulate(sorted_x))

    return (n + 1 - 2 * sum(cumx) / cumx[-1]) / n

ngrams

Modules:

Name Description
ngrams_base
ngrams_stats
Modules
ngrams_base

Modules:

Name Description
main
Classes Modules
main

Functions:

Name Description
ngrams

Generate n-grams from list of tokens.

serialize_ngram

Generates a string that uniquely represents an ngram

Classes Functions:
ngrams(tokens, min, max)

Generate n-grams from list of tokens.

Source code in src/cibmangotree/analyzers/ngrams/ngrams_base/main.py
504
505
506
507
508
509
510
def ngrams(tokens: list[str], min: int, max: int):
    """Generate n-grams from list of tokens."""
    for i in range(len(tokens) - min + 1):
        for n in range(min, max + 1):
            if i + n > len(tokens):
                break
            yield tokens[i : i + n]
serialize_ngram(ngram)

Generates a string that uniquely represents an ngram

Source code in src/cibmangotree/analyzers/ngrams/ngrams_base/main.py
513
514
515
def serialize_ngram(ngram: list[str]) -> str:
    """Generates a string that uniquely represents an ngram"""
    return " ".join(ngram)
ngrams_stats

Modules:

Name Description
main
Classes Modules
main
Classes

time_coordination

Modules:

Name Description
graph_utils

Weight-based graph cutting helpers.

time_coordination_base
Modules
graph_utils

Weight-based graph cutting helpers.

Ported from cib-time-analysis modules/my_graph.py — pure graph helpers only (no NiceGUI session / dataset-index wiring).

Functions:

Name Description
cut_graph_by_weight

Build a small graph of the strongest num_top links.

graph_from_edges_df

Rebuild a NetworkX graph from the primary edges parquet/table.

ranked_edges

Strongest edges first, capped at limit. Does not copy G.

remember_top_edges

Store the strongest links on G for Explore to slice.

Functions:
cut_graph_by_weight(G, num_top)

Build a small graph of the strongest num_top links.

Source code in src/cibmangotree/analyzers/time_coordination/graph_utils.py
41
42
43
44
45
46
47
48
def cut_graph_by_weight(G, num_top):
    """Build a small graph of the strongest ``num_top`` links."""
    ranked = G.graph.get(TOP_EDGES_KEY)
    chosen = ranked[:num_top] if ranked is not None else ranked_edges(G, num_top)
    cut = nx.Graph()
    for u, v, data in chosen:
        cut.add_edge(u, v, **data)
    return cut
graph_from_edges_df(edges_df)

Rebuild a NetworkX graph from the primary edges parquet/table.

Source code in src/cibmangotree/analyzers/time_coordination/graph_utils.py
51
52
53
54
55
56
57
58
59
60
61
def graph_from_edges_df(edges_df) -> nx.Graph:
    """Rebuild a NetworkX graph from the primary ``edges`` parquet/table."""
    G = nx.Graph()
    for row in edges_df.iter_rows(named=True):
        G.add_edge(
            row["user_a"],
            row["user_b"],
            weight=float(row["weight"]),
            norm_weight=float(row["norm_weight"]),
        )
    return G
ranked_edges(G, limit)

Strongest edges first, capped at limit. Does not copy G.

Source code in src/cibmangotree/analyzers/time_coordination/graph_utils.py
16
17
18
19
20
21
22
23
def ranked_edges(G, limit):
    """Strongest edges first, capped at limit. Does not copy G."""
    ranked = sorted(
        G.edges(data=True),
        key=lambda edge: edge[2].get("norm_weight", 0.0),
        reverse=True,
    )
    return ranked[:limit]
remember_top_edges(G, limit=EXPLORE_MAX_PAIRS, progress=None)

Store the strongest links on G for Explore to slice.

Source code in src/cibmangotree/analyzers/time_coordination/graph_utils.py
26
27
28
29
30
31
32
33
34
35
36
37
38
def remember_top_edges(G, limit=EXPLORE_MAX_PAIRS, progress=None):
    """Store the strongest links on ``G`` for Explore to slice."""
    total = G.number_of_edges()
    scanned = []
    for i, (u, v, data) in enumerate(G.edges(data=True), start=1):
        scanned.append((data.get("norm_weight", 0.0), u, v, data))
        if progress is not None and total and i % RANK_PROGRESS_EVERY == 0:
            progress.value = i / total
    scanned.sort(key=lambda item: item[0], reverse=True)
    G.graph[TOP_EDGES_KEY] = [(u, v, data) for _weight, u, v, data in scanned[:limit]]
    if progress is not None:
        progress.value = 1.0
    return G
time_coordination_base

Modules:

Name Description
analysis

Posting-time similarity graphs.

main

Primary entry: mapped posts → time-coordination graph → posts + edges parquet.

Classes Modules
analysis

Posting-time similarity graphs.

Ported from cib-time-analysis app/time_analysis.py. Dataset-button / hardcoded CSV path wiring is replaced by PreparedData (mapped input) or DatasetSpec (path-based, mainly for tests).

Classes:

Name Description
PreparedData

In-memory posts already sorted with rel_timestamp (mapped analyzer input).

Functions:

Name Description
ado_window

Score each post against the other accounts active in the following window.

content_for_user

Extract post text + timestamps for one user from an already-loaded frame.

get_tvec

Activity vector for one account.

overlapping_content_for_users

Posts from either user that fall within dt of at least one post by the other.

prepare_dataframe

Drop incomplete rows, add rel_timestamp, sort — same prep as read_csv.

time_overlap

Connect accounts whose binned activity overlaps.

Classes
PreparedData

Bases: NamedTuple

In-memory posts already sorted with rel_timestamp (mapped analyzer input).

Source code in src/cibmangotree/analyzers/time_coordination/time_coordination_base/analysis.py
40
41
42
43
44
45
46
class PreparedData(NamedTuple):
    """In-memory posts already sorted with ``rel_timestamp`` (mapped analyzer input)."""

    df: pd.DataFrame
    time_col_name: str
    user_col_name: str
    content_col_name: str
Functions:
ado_window(data_set, managed_progress_value, win_size=2, t_step=60, phase=None)

Score each post against the other accounts active in the following window.

left and right index the sorted posts. in_window holds how many posts each account has between those indexes. Edge weight is that count; norm_weight divides it by the number of posts in the slice.

Source code in src/cibmangotree/analyzers/time_coordination/time_coordination_base/analysis.py
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
def ado_window(data_set, managed_progress_value, win_size=2, t_step=60, phase=None):
    """Score each post against the other accounts active in the following window.

    ``left`` and ``right`` index the sorted posts. ``in_window`` holds how many
    posts each account has between those indexes. Edge weight is that count;
    norm_weight divides it by the number of posts in the slice.
    """
    dataset, csv_df, _time_data = _load_dataset(data_set, managed_progress_value, phase)
    user_col_name = dataset.user_col_name
    A = nx.Graph()
    dt = int(win_size * t_step)
    times = csv_df["rel_timestamp"].to_numpy()
    users = csv_df[user_col_name].to_numpy()
    n_posts = len(times)
    in_window: dict = {}
    left = 0
    right = 0

    with _timed():
        for ipost in range(n_posts):
            if ipost % PROGRESS_EVERY_N_POSTS == 0:
                print(f"Processing post {ipost+1}/{n_posts}")
                managed_progress_value.value = (ipost + 1) / n_posts

            post_time = times[ipost]
            stop = post_time + dt
            while right < n_posts and times[right] <= stop:
                arriving = users[right]
                in_window[arriving] = in_window.get(arriving, 0) + 1
                right += 1
            while left < ipost and times[left] < post_time:
                leaving = users[left]
                in_window[leaving] -= 1
                if in_window[leaving] == 0:
                    del in_window[leaving]
                left += 1

            num_posts = right - left
            user = users[ipost]
            for other_user, w in in_window.items():
                if other_user == user:
                    continue
                if A.get_edge_data(user, other_user) is None:
                    A.add_edge(user, other_user, weight=w, norm_weight=w / num_posts)
                else:
                    A[user][other_user]["weight"] += w
                    A[user][other_user]["norm_weight"] += w / num_posts

    return A
content_for_user(user, df, user_col_name, content_col_name, time_col_name)

Extract post text + timestamps for one user from an already-loaded frame.

Source code in src/cibmangotree/analyzers/time_coordination/time_coordination_base/analysis.py
253
254
255
256
257
258
259
def content_for_user(user, df, user_col_name, content_col_name, time_col_name):
    """Extract post text + timestamps for one user from an already-loaded frame."""
    user_rows = df[df[user_col_name] == user]
    return (
        user_rows[content_col_name].to_numpy(),
        user_rows[time_col_name].to_numpy(),
    )
get_tvec(user, df, user_col_name, t_step=1)

Activity vector for one account.

Marks each second that has a post, then sums those marks into bins of t_step seconds. time_overlap matches this binning in _activity_matrix and does not build this per-second vector per account.

Source code in src/cibmangotree/analyzers/time_coordination/time_coordination_base/analysis.py
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
def get_tvec(user, df, user_col_name, t_step=1):
    """Activity vector for one account.

    Marks each second that has a post, then sums those marks into bins of
    ``t_step`` seconds. ``time_overlap`` matches this binning in
    ``_activity_matrix`` and does not build this per-second vector per account.
    """
    post_times = df[df[user_col_name] == user]["rel_timestamp"].to_numpy(dtype=int)
    num_posts = len(post_times)
    t_end = int(np.max(df["rel_timestamp"])) + 1
    tvec = np.zeros(t_end)
    tvec[post_times] = 1
    if t_step > 1:
        new_len = int(np.ceil(t_end / t_step) * t_step)
        new_vec = np.zeros(new_len)
        new_vec[: len(tvec)] = tvec
        tvec = np.sum(new_vec.reshape(-1, t_step), axis=1)
    return tvec, num_posts
overlapping_content_for_users(user0, user1, df, user_col_name, content_col_name, time_col_name, dt)

Posts from either user that fall within dt of at least one post by the other.

Returns a list of (user, content, display_time) sorted by relative timestamp.

Source code in src/cibmangotree/analyzers/time_coordination/time_coordination_base/analysis.py
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
def overlapping_content_for_users(
    user0,
    user1,
    df,
    user_col_name,
    content_col_name,
    time_col_name,
    dt,
):
    """Posts from either user that fall within dt of at least one post by the other.

    Returns a list of (user, content, display_time) sorted by relative timestamp.
    """
    rows0 = df[df[user_col_name] == user0]
    rows1 = df[df[user_col_name] == user1]
    rel0 = rows0["rel_timestamp"].to_numpy(dtype=float)
    rel1 = rows1["rel_timestamp"].to_numpy(dtype=float)
    mask0 = _mask_within_dt(rel0, rel1, dt)
    mask1 = _mask_within_dt(rel1, rel0, dt)

    items: list[tuple] = []
    for keep, content, stamp, rel in zip(
        mask0,
        rows0[content_col_name].to_numpy(),
        rows0[time_col_name].to_numpy(),
        rel0,
        strict=True,
    ):
        if keep:
            items.append((user0, content, stamp, rel))
    for keep, content, stamp, rel in zip(
        mask1,
        rows1[content_col_name].to_numpy(),
        rows1[time_col_name].to_numpy(),
        rel1,
        strict=True,
    ):
        if keep:
            items.append((user1, content, stamp, rel))

    items.sort(key=lambda row: row[3])
    return [(user, content, stamp) for user, content, stamp, _rel in items]
prepare_dataframe(df, time_col_name, user_col_name, content_col_name, progress=None)

Drop incomplete rows, add rel_timestamp, sort — same prep as read_csv.

Source code in src/cibmangotree/analyzers/time_coordination/time_coordination_base/analysis.py
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
def prepare_dataframe(
    df: pd.DataFrame,
    time_col_name: str,
    user_col_name: str,
    content_col_name: str,
    progress=None,
) -> PreparedData:
    """Drop incomplete rows, add ``rel_timestamp``, sort — same prep as ``read_csv``."""
    csv_df = df.dropna(subset=[user_col_name, time_col_name, content_col_name]).copy()
    series = csv_df[time_col_name]
    n_rows = len(csv_df)
    time_data: list[float] = []

    if pd.api.types.is_datetime64_any_dtype(series):
        # Normalize any datetime resolution to unix seconds.
        values = (
            pd.to_datetime(series, utc=True)
            .map(lambda t: t.timestamp())
            .to_numpy(dtype=float)
        )
        for i, ts in enumerate(values):
            time_data.append(float(ts))
            if (
                progress is not None
                and n_rows
                and (i % READ_PROGRESS_EVERY == 0 or i + 1 == n_rows)
            ):
                progress.value = (i + 1) / n_rows
    else:
        for i, row in enumerate(series):
            if hasattr(row, "timestamp"):
                timestamp = float(row.timestamp())
            else:
                timestamp = time.mktime(parser.parse(str(row)).timetuple())
            time_data.append(timestamp)
            if (
                progress is not None
                and n_rows
                and (i % READ_PROGRESS_EVERY == 0 or i + 1 == n_rows)
            ):
                progress.value = (i + 1) / n_rows

    time_arr = np.array(time_data)
    csv_df["rel_timestamp"] = time_arr - np.min(time_arr)
    csv_df = csv_df.sort_values("rel_timestamp")
    return PreparedData(
        df=csv_df,
        time_col_name=time_col_name,
        user_col_name=user_col_name,
        content_col_name=content_col_name,
    )
time_overlap(data_set, managed_progress_value, win_size=2, t_step=60, min_posts=DEFAULT_MIN_POSTS, phase=None)

Connect accounts whose binned activity overlaps.

Source code in src/cibmangotree/analyzers/time_coordination/time_coordination_base/analysis.py
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
def time_overlap(
    data_set,
    managed_progress_value,
    win_size=2,
    t_step=60,
    min_posts=DEFAULT_MIN_POSTS,
    phase=None,
):
    """Connect accounts whose binned activity overlaps."""
    dataset, csv_df, _time_data = _load_dataset(data_set, managed_progress_value, phase)
    user_col_name = dataset.user_col_name
    dt = int(win_size * t_step)
    H = nx.Graph()
    names, timelines, num_posts = _activity_matrix(
        csv_df, user_col_name, t_step, min_posts
    )
    if not names:
        managed_progress_value.value = 1.0
        return H
    H.add_nodes_from(names)

    n_users = len(names)
    chunk = 64
    with _timed():
        for start in range(0, n_users, chunk):
            stop = min(start + chunk, n_users)
            print(f"Processing user {start}/{n_users}")
            smoothed = num_posts[start:stop, None] * _moving_sum_same(
                timelines[start:stop], dt
            )
            scores = smoothed @ timelines.T
            for local, i in enumerate(range(start, stop)):
                for j in range(i + 1, n_users):
                    overlap = float(scores[local, j])
                    if overlap > MIN_OVERLAP:
                        H.add_edge(
                            names[i],
                            names[j],
                            weight=overlap,
                            norm_weight=overlap,
                        )
            managed_progress_value.value = stop / n_users

    return H
main

Primary entry: mapped posts → time-coordination graph → posts + edges parquet.

Classes Functions: