Skip to content

fix: support Telegram Desktop full exports (#188) - #226

Open
Yu-0312 wants to merge 1 commit into
xming521:masterfrom
Yu-0312:fix/issue-188-telegram-export
Open

fix: support Telegram Desktop full exports (#188)#226
Yu-0312 wants to merge 1 commit into
xming521:masterfrom
Yu-0312:fix/issue-188-telegram-export

Conversation

@Yu-0312

@Yu-0312 Yu-0312 commented Aug 22, 2026

Copy link
Copy Markdown

Closes #188. Support Telegram Desktop full exports with chats.list while preserving compatibility with individual chat exports. Add regression tests for both formats.

Sourcery 摘要

支持 Telegram Desktop 完整导出和单个聊天导出,并为每个聊天生成独立的 CSV 数据集。

新功能:

  • 支持处理包含多个聊天的 Telegram Desktop 完整导出,其中聊天数据位于 chats.list 中。
  • 继续支持顶层包含聊天数据的单个 Telegram 聊天导出。

增强功能:

  • 为每个导出的聊天生成独立且经过清理的 CSV 输出路径,并跳过不包含结果文件的文件夹。

测试:

  • 增加对 Telegram 完整导出和单个聊天导出的回归测试覆盖。
Original summary in English

Summary by Sourcery

Support both Telegram Desktop full exports and individual chat exports while producing separate CSV datasets for each chat.

New Features:

  • Support processing Telegram Desktop full exports containing multiple chats in chats.list.
  • Continue supporting individual Telegram chat exports with top-level chat data.

Enhancements:

  • Generate separate sanitized CSV output paths for each exported chat and skip folders without a result file.

Tests:

  • Add regression coverage for full Telegram exports and individual chat exports.

@sourcery-ai

sourcery-ai Bot commented Aug 22, 2026

Copy link
Copy Markdown

审查者指南

新增对 Telegram Desktop 完整导出(chats.list)的支持,同时保持与单独聊天导出的兼容性,并为两种导出格式引入回归测试。解析器现在会遍历完整导出中的聊天,将聊天元数据规范化为安全的路径组件,并为每个聊天创建单独的 CSV 输出文件。

Telegram 导出处理的时序图

sequenceDiagram
    participant Export as TelegramExport
    participant Processor as process_telegram_dataset
    participant Iterator as _iter_export_chats
    participant Parser as TelegramChatParser
    participant CSV as CSVOutput

    Export->>Processor: load result.json
    Processor->>Iterator: _iter_export_chats(jdata)
    alt full export
        Iterator-->>Processor: chats.list chat objects
    else individual chat export
        Iterator-->>Processor: top-level chat object
    end
    loop each chat
        Processor->>Parser: process_chat(chat_data)
        Parser-->>Processor: messages
        Processor->>CSV: to_csv(messages, csv_file_path)
        Processor->>Parser: copy_received_images(messages, folder_path)
    end
Loading

文件级变更

变更 详细信息 文件
规范化 Telegram 导出布局,使解析器同时支持完整导出(chats.list)和单独聊天导出。
  • 引入 _iter_export_chats,从完整导出结构(chats.list)或顶层为单独聊天的 JSON 中生成聊天字典,使后续处理流程能够使用统一的聊天结构。
  • 更新 process_telegram_dataset,将每个子文件夹视为一个可能包含多个聊天的 Telegram 导出,通过 _iter_export_chats 获取聊天列表,并使用索引遍历这些聊天。
  • 调整日志路径:对于缺少 result.json 的文件夹,跳过处理并记录警告;对于不包含有效消息的聊天,记录更具体的警告,其中包括聊天名称和文件夹。
weclone/data/chat_parsers/telegram_parser.py
使用经过清理并带有回退值的路径组件,增强 CSV 文件夹名和文件名的构建逻辑。
  • 添加 _safe_path_component 辅助函数,用于移除路径组件中的不安全字符,仅允许字母数字字符以及点号、下划线和连字符;如果清理结果为空,则使用默认标签。
  • 当导出数据缺少名称或 ID 时,使用基于聊天索引的回退值,确保完整导出中的所有聊天都能生成确定且符合文件系统安全要求的 CSV 文件夹名称。
weclone/data/chat_parsers/telegram_parser.py
添加覆盖 Telegram Desktop 完整导出和单独聊天导出的回归测试。
  • 创建 test_process_full_telegram_export,在 dataset/telegram/ChatExport/result.json 中构造包含多个聊天的完整导出(chats.list),并断言每个聊天的 CSV 文件都已创建且内容符合预期。
  • 创建 test_process_individual_chat_export,在 dataset/telegram/SingleChat 下构造单独聊天的 result.json,并验证旧版单独聊天处理路径仍能生成预期的 CSV 输出。
  • 提供轻量级 make_config 辅助函数,用于构造 TelegramChatParser 测试所需的最小配置对象(包括 telegram_args.my_id),并使用 tmp_path+monkeypatch.chdir 针对临时数据集布局运行测试。
tests/test_telegram_parser.py

针对关联 Issue 的评估

Issue 目标 已解决 说明
#188 支持解析 Telegram Desktop 完整导出的 result.json,其中聊天记录位于 chats.list 中,并为其中的各个聊天生成可用的数据集 CSV。
#188 保留对单独聊天导出格式(聊天信息和 messages 位于 result.json 顶层)的兼容性。
#188 在处理 Telegram 导出时正确提取文本消息,并忽略不需要处理的其他导出内容,同时避免因聊天名称、类型或 ID 导致输出路径无效。

可能关联的 Issue


提示和命令

与 Sourcery 交互

  • 触发新的审查: 在拉取请求中评论 @sourcery-ai review
  • 继续讨论: 直接回复 Sourcery 的审查评论。
  • 根据审查评论生成 GitHub Issue: 回复审查评论,请 Sourcery 根据该评论创建 Issue。你也可以使用 @sourcery-ai issue 回复审查评论,以根据该评论创建 Issue。
  • 生成拉取请求标题: 在拉取请求标题的任意位置写入 @sourcery-ai,即可随时生成标题。你也可以在拉取请求中评论 @sourcery-ai title,以随时(重新)生成标题。
  • 生成拉取请求摘要: 在拉取请求正文的任意位置写入 @sourcery-ai summary,即可在指定位置随时生成 PR 摘要。你也可以在拉取请求中评论 @sourcery-ai summary,以随时(重新)生成摘要。
  • 生成审查者指南: 在拉取请求中评论 @sourcery-ai guide,即可随时(重新)生成审查者指南。
  • 解决所有 Sourcery 评论: 在拉取请求中评论 @sourcery-ai resolve,即可解决所有 Sourcery 评论。如果你已经处理完所有评论且不想再看到它们,此功能会很有用。
  • 忽略所有 Sourcery 审查: 在拉取请求中评论 @sourcery-ai dismiss,即可忽略所有现有的 Sourcery 审查。如果你想从新的审查开始,这尤其有用——别忘了评论 @sourcery-ai review 来触发新的审查!

自定义使用体验

访问你的 仪表板

  • 启用或禁用审查功能,例如 Sourcery 生成的拉取请求摘要、审查者指南等。
  • 更改审查语言。
  • 添加、删除或编辑自定义审查说明。
  • 调整其他审查设置。

获取帮助

Original review guide in English

Reviewer's Guide

Adds support for Telegram Desktop full exports (chats.list) to the Telegram parser while maintaining compatibility with single-chat exports, and introduces regression tests for both export formats. The parser now iterates chats within a full export, normalizes chat metadata into safe path components, and creates separate CSV outputs per chat.

Sequence diagram for Telegram export processing

sequenceDiagram
    participant Export as TelegramExport
    participant Processor as process_telegram_dataset
    participant Iterator as _iter_export_chats
    participant Parser as TelegramChatParser
    participant CSV as CSVOutput

    Export->>Processor: load result.json
    Processor->>Iterator: _iter_export_chats(jdata)
    alt full export
        Iterator-->>Processor: chats.list chat objects
    else individual chat export
        Iterator-->>Processor: top-level chat object
    end
    loop each chat
        Processor->>Parser: process_chat(chat_data)
        Parser-->>Processor: messages
        Processor->>CSV: to_csv(messages, csv_file_path)
        Processor->>Parser: copy_received_images(messages, folder_path)
    end
Loading

File-Level Changes

Change Details Files
Normalize Telegram export layouts so the parser supports both full exports (chats.list) and individual chat exports.
  • Introduce _iter_export_chats to yield chat dicts from either a full export structure (chats.list) or a single-chat top-level JSON, allowing the rest of the pipeline to work with a unified chat shape.
  • Update process_telegram_dataset to treat each subfolder as a Telegram export that may contain multiple chats, materializing the list of chats via _iter_export_chats and iterating them with an index.
  • Adjust the logging path so that folders missing result.json are skipped with a warning and chats without valid messages log a more specific warning including the chat name and folder.
weclone/data/chat_parsers/telegram_parser.py
Harden CSV folder/file name construction using sanitized path components with fallbacks.
  • Add _safe_path_component helper to strip unsafe characters from path components, allowing only alphanumeric characters plus dot, underscore, and dash, and falling back to a default label if the result is empty.
  • Use chat index–based fallbacks for name/id when the export is missing those fields, ensuring deterministic and filesystem-safe CSV folder names for all chats in a full export.
weclone/data/chat_parsers/telegram_parser.py
Add regression tests covering both full Telegram Desktop exports and individual chat exports.
  • Create test_process_full_telegram_export which synthesizes a full export in dataset/telegram/ChatExport/result.json with chats.list containing multiple chats and asserts that per-chat CSV files are created with the expected content.
  • Create test_process_individual_chat_export which synthesizes a single-chat result.json under dataset/telegram/SingleChat and verifies that the legacy single-chat path still produces the expected CSV output.
  • Provide a lightweight make_config helper that constructs a minimal config object (including telegram_args.my_id) needed by TelegramChatParser for tests, and use tmp_path+monkeypatch.chdir to run tests against a temporary dataset layout.
tests/test_telegram_parser.py

Assessment against linked issues

Issue Objective Addressed Explanation
#188 支持解析 Telegram Desktop 完整导出的 result.json,其中聊天记录位于 chats.list 中,并为其中的各个聊天生成可用的数据集 CSV。
#188 保留对单独聊天导出格式(聊天信息和 messages 位于 result.json 顶层)的兼容性。
#188 在处理 Telegram 导出时正确提取文本消息,并忽略不需要处理的其他导出内容,同时避免因聊天名称、类型或 ID 导致输出路径无效。

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你好——我已经审阅了你的更改,看起来非常棒!


Sourcery 对开源项目免费——如果你喜欢我们的审查结果,请考虑分享它们 ✨
请帮我变得更有用!请在每条评论上点击 👍 或 👎,我会利用这些反馈来改进审查结果。
Original comment in English

Hey - I've reviewed your changes and they look great!


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telegram 支持从设置导出的聊天记录

1 participant