# chatpack: How I Compressed 11 Million Tokens Down to 850K for Chat Analysis *From manual scripts to automation: A Rust CLI tool for preparing chat exports for LLM analysis* --- ## The Problem Nobody Notices Gemini refused to analyze my chat. *"Failed to generate content. Please try again."* And I needed to understand what patterns exist in how our community communicates. I was trying to analyze a private developer group at a programming school where I study. I needed to identify trends in how students communicate and what topics come up most often. I needed to identify trends: why do so few students make it to the end? What changed when the platform was updated? What advice did graduates give? I exported the chat from Telegram. 34 thousand messages, three years of history. Opened `result.json` - 26 megabytes. The problem isn't the file size. The problem is that 80% of those 26 megabytes is noise: JSON brackets, `"type": "message"` keys on every line, metadata about stickers you don't need. Before this, I solved similar tasks with scripts that searched for keywords to find relevant sections. I sorted by nicknames and dates. The most annoying part - everything had to be done manually, and some data got lost due to this crude initial filtering. That's how [chatpack](https://github.com/berektassuly/chatpack) was born. --- ## What chatpack Does ```plaintext Telegram JSON (11M tokens) ↓ [chatpack] ↓ CSV (850K tokens) → LLM → Insights ``` One command: ```bash chatpack tg result.json -o context.csv ``` Result: * **Input:** 11.2M tokens (raw Telegram JSON) * **Output:** 850K tokens (optimized CSV) * **Compression:** 13x Now you can upload this CSV to LLM with an actual prompt: ```plaintext Here's our team's chat history for 2023. 1. Identify the top 5 most discussed topics 2. Who was most active in each quarter? 3. Is there a correlation between time of day and message tone? ``` ### Supported Platforms ```bash chatpack tg chat.json # Telegram chatpack wa chat.txt # WhatsApp chatpack ig message_1.json # Instagram chatpack dc export.json # Discord ``` I started with Telegram work groups. Then I added other messengers - to automate my processes and identify common trends. --- ## Why CSV Compresses Best Comparison on real data (34K messages): | Format | Tokens | Compression | | --- | --- | --- | | Raw JSON | 11,177,258 | baseline | | **CSV** | 849,915 | **13.2x** | | JSONL | 1,029,130 | 10.9x | | JSON (clean) | 1,333,586 | 8.4x | Why such a difference? ```json // JSON: each message is {"sender": "Alice", "content": "Hello"} // CSV: just data Alice;Hello ``` JSON wastes tokens on: * Brackets `{}` and `[]` * Keys `"sender":`, `"content":` - for every message * Quotes around every string CSV uses `;` as a delimiter (one character) and writes keys only in the header. You can also work with CSV in Excel - calculate response times, average message length, build engagement charts. --- ## Merge Consecutive: The Obvious Feature Everyone Ignores A typical chat looks like this: ```plaintext Alice: Hey Alice: How are you? Alice: Did you see the new project? Bob: Yeah, I looked Bob: Pretty good overall ``` After merge: ```plaintext Alice: Hey How are you? Did you see the new project? Bob: Yeah, I looked Pretty good overall ``` Here's what the actual CSV output looks like: ```plaintext # Without merge (5 rows, ~45 tokens) Sender;Content Alice;Hey Alice;How are you? Alice;Did you see the new project? Bob;Yeah, I looked Bob;Pretty good overall # With merge (2 rows, ~35 tokens) Sender;Content Alice;Hey How are you? Did you see the new project? Bob;Yeah, I looked Pretty good overall ``` **5 messages → 2 entries.** On 34K messages, this gives 24% reduction before even choosing a format. But the main benefit isn't tokens. Embedding a single complete thought is processed much better than 5 fragmented pieces of the same thought. For RAG pipelines, this is critical. Anyone who has worked with large chats eventually arrives at merge. This isn't an insight - it's common knowledge among those who analyze conversations. --- ## Why Not Just Use jq or a Python Script? You could write `jq '.messages[] | {from, text}'`, but that won't give you merge, date filters, or fix Mojibake in Instagram exports. chatpack handles all the edge cases so you don't have to. --- ## Architecture: Why Traits, Not If-Else When I wrote the Telegram parser, I built in extensibility from the start: ```rust pub trait ChatParser: Send + Sync { fn name(&self) -> &'static str; fn parse(&self, file_path: &str) -> Result, Box>; } ``` Adding a new messenger - three steps: ```rust // 1. Implement the parser pub struct SignalParser; impl ChatParser for SignalParser { fn name(&self) -> &'static str { "Signal" } fn parse(&self, path: &str) -> Result, _> { // ... } } // 2. Add to Source enum pub enum Source { Telegram, WhatsApp, Instagram, Discord, Signal } // 3. Add to factory Source::Signal => Box::new(SignalParser::new()) ``` This was a planned decision from day one. The Strategy pattern isn't overengineering when you know you'll be extending. --- ## Practical Use Cases ### Engagement Analysis I needed to add an engagement metric - to identify when a person is most active and how this changed over the years. ```bash chatpack tg group.json -t -o timeline.csv ``` The `-t` flag adds timestamps. Now the CSV has a date column for each message: ```plaintext Timestamp;Sender;Content 2024-01-15 10:30:00;Alice;Good morning! 2024-01-15 10:31:00;Bob;Hey ``` Upload to Claude: *"Build an activity chart by hour for each participant."* An LLM can't understand what someone is doing outside the chat. But general trends - when the platform changed, who was active during which period, how team mood shifted - it identifies perfectly. ### HR Patterns I use filters often when I need to identify patterns for a specific person: ```bash chatpack tg chat.json --from "Ivan" -o ivan.csv ``` I get all messages from one participant. Then analyze: * Grammar * Humor style * Average message length * Activity times It's like HR screening before team selection - just automated. ### Project Archaeology The most useful case for me personally: ```bash chatpack tg chat.json --after 2022-01-01 --before 2023-01-01 -o year_2022.csv ``` I was looking for information about the old platform, former staff and students, advice from graduates about where they got jobs. The LLM handled the task, identifying what I needed. Over time, I formulated concrete ideas for improving the platform. --- ## Technical Details ### Performance * **Speed:** 20-50K messages/sec * **Memory:** ~3x file size (entire JSON in memory) * **Ceiling:** files up to 500MB comfortably, 1GB+ needs streaming (not implemented) Why no streaming? I just needed a library that would solve my problem. Most of the time, message exports don't exceed 1 MB. I consciously chose not to overcomplicate things. ### When chatpack Won't Help * **Files >1GB** - no streaming, will eat your RAM * **Media context needed** - photos, voice messages are ignored, text only * **Real-time analysis** - this is a batch tool * **Preserving exact JSON structure** - chatpack normalizes everything to a flat format ### Why Rust Rust is currently my main stack. I used it because the project fits the criteria: * CLI tool - Rust is ideal * Need speed - check * Need reliability - types prevent mistakes Thanks to this project, I also learned how to work with WASM and built a web version. ### Handling Edge Cases WhatsApp exports dates in different formats depending on locale: ```plaintext [1/15/24, 10:30 AM] // US [15.01.24, 10:30] // EU 26.10.2025, 20:40 - // RU ``` chatpack auto-detects the format from the first 20 lines and parses correctly. Instagram stores UTF-8 text as ISO-8859-1 (Mojibake). "Привет" becomes "Привет". The parser fixes this automatically. --- ## WASM: chatpack in the Browser Not everyone wants to install a CLI. Some people just need to convert one file without touching the terminal. That's why there's [chatpack.berektassuly.com](https://chatpack.berektassuly.com). The workflow is simple: 1. Drag & drop your export file 2. Choose output format 3. Download the result The key point: **your file never leaves your machine**. Everything is processed locally in the browser via WebAssembly. No server, no uploads, no privacy concerns. Repository: [github.com/Berektassuly/chatpack-web](https://github.com/Berektassuly/chatpack-web) --- ## Installation and Usage ### Pre-built Binaries Download for your platform from [GitHub Releases](https://github.com/berektassuly/chatpack/releases/latest): * Windows (x64) * macOS (Intel and Apple Silicon) * Linux (x64) ### Via Cargo ```bash cargo install chatpack ``` ### As a Library ```toml [dependencies] chatpack = "0.3" ``` ```rust use chatpack::prelude::*; let parser = create_parser(Source::Telegram); let messages = parser.parse("chat.json")?; let merged = merge_consecutive(messages); write_csv(&merged, "output.csv", &OutputConfig::new())?; ``` --- ## Conclusion chatpack is a tool I wrote for myself. The problem was specific: analyze chat history through an LLM. The solution turned out to be universal. If I were starting today, I'd keep the development process the same: 1. Telegram parser 2. Trait for extensibility 3. Other messengers as needed 4. WASM for those who find CLI overkill If anyone needs it - feel free to use it in your project. If there are issues, I'll listen to what users need and fix it. --- **Links:** * GitHub: [github.com/berektassuly/chatpack](https://github.com/berektassuly/chatpack) * Web: [chatpack.berektassuly.com](https://chatpack.berektassuly.com) * WASM repo: [github.com/Berektassuly/chatpack-web](https://github.com/Berektassuly/chatpack-web) * Author: [Mukhammedali Berektassuly](https://www.linkedin.com/in/mukhammedali-berektassuly/)