The Problem Nobody Notices

Gemini refused to analyze my chat. "Failed to generate content. Please try again." And I needed to understand what patterns exist in how our community communicates.

I was trying to analyze a private developer group at a programming school where I study. I needed to identify trends in how students communicate and what topics come up most often. I needed to identify trends: why do so few students make it to the end? What changed when the platform was updated? What advice did graduates give?

I exported the chat from Telegram. 34 thousand messages, three years of history. Opened result.json - 26 megabytes. The problem isn't the file size. The problem is that 80% of those 26 megabytes is noise: JSON brackets, "type": "message" keys on every line, metadata about stickers you don't need.

Before this, I solved similar tasks with scripts that searched for keywords to find relevant sections. I sorted by nicknames and dates. The most annoying part - everything had to be done manually, and some data got lost due to this crude initial filtering.

That's how chatpack was born.

What chatpack Does

Telegram JSON (11M tokens)
        ↓
    [chatpack]
        ↓
   CSV (850K tokens) → LLM → Insights

One command:

chatpack tg result.json -o context.csv

Result:

• Input: 11.2M tokens (raw Telegram JSON)

• Output: 850K tokens (optimized CSV)

• Compression: 13x

Now you can upload this CSV to LLM with an actual prompt:

Here's our team's chat history for 2023.

1. Identify the top 5 most discussed topics
2. Who was most active in each quarter?
3. Is there a correlation between time of day and message tone?

Supported Platforms

chatpack tg chat.json      # Telegram
chatpack wa chat.txt       # WhatsApp
chatpack ig message_1.json # Instagram
chatpack dc export.json    # Discord

I started with Telegram work groups. Then I added other messengers - to automate my processes and identify common trends.

Why CSV Compresses Best

Comparison on real data (34K messages):

Why such a difference?

// JSON: each message is
{"sender": "Alice", "content": "Hello"}

// CSV: just data
Alice;Hello

JSON wastes tokens on:

• Brackets {} and []

• Keys "sender":, "content": - for every message

• Quotes around every string

CSV uses ; as a delimiter (one character) and writes keys only in the header.

You can also work with CSV in Excel - calculate response times, average message length, build engagement charts.

Merge Consecutive: The Obvious Feature Everyone Ignores

A typical chat looks like this:

Alice: Hey
Alice: How are you?
Alice: Did you see the new project?
Bob: Yeah, I looked
Bob: Pretty good overall

After merge:

Alice: Hey
How are you?
Did you see the new project?
Bob: Yeah, I looked
Pretty good overall

Here's what the actual CSV output looks like:

# Without merge (5 rows, ~45 tokens)
Sender;Content
Alice;Hey
Alice;How are you?
Alice;Did you see the new project?
Bob;Yeah, I looked
Bob;Pretty good overall

# With merge (2 rows, ~35 tokens)
Sender;Content
Alice;Hey
How are you?
Did you see the new project?
Bob;Yeah, I looked
Pretty good overall

5 messages → 2 entries. On 34K messages, this gives 24% reduction before even choosing a format.

But the main benefit isn't tokens. Embedding a single complete thought is processed much better than 5 fragmented pieces of the same thought. For RAG pipelines, this is critical.

Anyone who has worked with large chats eventually arrives at merge. This isn't an insight - it's common knowledge among those who analyze conversations.

Why Not Just Use jq or a Python Script?

You could write jq '.messages[] | {from, text}', but that won't give you merge, date filters, or fix Mojibake in Instagram exports. chatpack handles all the edge cases so you don't have to.

Architecture: Why Traits, Not If-Else

When I wrote the Telegram parser, I built in extensibility from the start:

pub trait ChatParser: Send + Sync {
    fn name(&self) -> &'static str;
    fn parse(&self, file_path: &str) -> Result<Vec<InternalMessage>, Box<dyn Error>>;
}

Adding a new messenger - three steps:

// 1. Implement the parser
pub struct SignalParser;
impl ChatParser for SignalParser {
    fn name(&self) -> &'static str { "Signal" }
    fn parse(&self, path: &str) -> Result<Vec<InternalMessage>, _> {
        // ...
    }
}

// 2. Add to Source enum
pub enum Source { Telegram, WhatsApp, Instagram, Discord, Signal }

// 3. Add to factory
Source::Signal => Box::new(SignalParser::new())

This was a planned decision from day one. The Strategy pattern isn't overengineering when you know you'll be extending.

Practical Use Cases

Engagement Analysis

I needed to add an engagement metric - to identify when a person is most active and how this changed over the years.

chatpack tg group.json -t -o timeline.csv

The -t flag adds timestamps. Now the CSV has a date column for each message:

Timestamp;Sender;Content
2024-01-15 10:30:00;Alice;Good morning!
2024-01-15 10:31:00;Bob;Hey

Upload to Claude: "Build an activity chart by hour for each participant."

An LLM can't understand what someone is doing outside the chat. But general trends - when the platform changed, who was active during which period, how team mood shifted - it identifies perfectly.

HR Patterns

I use filters often when I need to identify patterns for a specific person:

chatpack tg chat.json --from "Ivan" -o ivan.csv

I get all messages from one participant. Then analyze:

• Grammar

• Humor style

• Average message length

• Activity times

It's like HR screening before team selection - just automated.

Project Archaeology

The most useful case for me personally:

chatpack tg chat.json --after 2022-01-01 --before 2023-01-01 -o year_2022.csv

I was looking for information about the old platform, former staff and students, advice from graduates about where they got jobs. The LLM handled the task, identifying what I needed. Over time, I formulated concrete ideas for improving the platform.

Technical Details

Performance

• Speed: 20-50K messages/sec

• Memory: ~3x file size (entire JSON in memory)

• Ceiling: files up to 500MB comfortably, 1GB+ needs streaming (not implemented)

Why no streaming? I just needed a library that would solve my problem. Most of the time, message exports don't exceed 1 MB. I consciously chose not to overcomplicate things.

When chatpack Won't Help

• Files >1GB - no streaming, will eat your RAM

• Media context needed - photos, voice messages are ignored, text only

• Real-time analysis - this is a batch tool

• Preserving exact JSON structure - chatpack normalizes everything to a flat format

Why Rust

Rust is currently my main stack. I used it because the project fits the criteria:

• CLI tool - Rust is ideal

• Need speed - check

• Need reliability - types prevent mistakes

Thanks to this project, I also learned how to work with WASM and built a web version.

Handling Edge Cases

WhatsApp exports dates in different formats depending on locale:

[1/15/24, 10:30 AM]      // US
[15.01.24, 10:30]        // EU
26.10.2025, 20:40 -      // RU

chatpack auto-detects the format from the first 20 lines and parses correctly.

Instagram stores UTF-8 text as ISO-8859-1 (Mojibake). "Привет" becomes "Привет". The parser fixes this automatically.

WASM: chatpack in the Browser

Not everyone wants to install a CLI. Some people just need to convert one file without touching the terminal. That's why there's chatpack.berektassuly.com.

The workflow is simple:

1. Drag & drop your export file

2. Choose output format

3. Download the result

The key point: your file never leaves your machine. Everything is processed locally in the browser via WebAssembly. No server, no uploads, no privacy concerns.

Repository: github.com/Berektassuly/chatpack-web

Installation and Usage

Pre-built Binaries

Download for your platform from GitHub Releases:

• Windows (x64)

• macOS (Intel and Apple Silicon)

• Linux (x64)

Via Cargo

cargo install chatpack

As a Library

[dependencies]
chatpack = "0.3"

use chatpack::prelude::*;

let parser = create_parser(Source::Telegram);
let messages = parser.parse("chat.json")?;
let merged = merge_consecutive(messages);
write_csv(&merged, "output.csv", &OutputConfig::new())?;

Conclusion

chatpack is a tool I wrote for myself. The problem was specific: analyze chat history through an LLM. The solution turned out to be universal.

If I were starting today, I'd keep the development process the same:

1. Telegram parser

2. Trait for extensibility

3. Other messengers as needed

4. WASM for those who find CLI overkill

If anyone needs it - feel free to use it in your project. If there are issues, I'll listen to what users need and fix it.

Links:

• GitHub: github.com/berektassuly/chatpack

• Web: chatpack.berektassuly.com

• WASM repo: github.com/Berektassuly/chatpack-web

• Author: Mukhammedali Berektassuly

Imagine you're building a service for issuing digital diplomas that records document hashes on the Solana blockchain. Everything works great until you try to write your first unit test. Suddenly, you realize that testing simple business logic requires spinning up a local Solana validator, obtaining test tokens, and praying the network doesn't crash in the middle of your CI/CD pipeline. And what happens when your client asks for Ethereum support tomorrow? Rewrite half your codebase?

In this article, I'll show you how we solved this problem in a real production project using the power of Rust's type system and the Strategy pattern. You'll learn how to properly leverage async-trait, why dyn Trait sometimes beats generics, and how to write tests that run in milliseconds instead of minutes.

The Anti-Pattern: How We Suffered BEFORE Refactoring

Let's be honest - our first version looked something like this:

// ❌ BAD: Direct dependency on Solana everywhere
use solana_client::rpc_client::RpcClient;
use solana_sdk::{signature::Keypair, transaction::Transaction};

pub struct AppState {
    pub solana_client: RpcClient,  // <- Concrete implementation!
    pub keypair: Keypair,
    pub db_client: Postgrest,
}

pub async fn issue_diploma(
    State(state): State<Arc<AppState>>,
    // ...
) -> Result<Json<IssueResponse>, AppError> {
    // Business logic tightly coupled to Solana
    let instruction = /* create Solana-specific instruction */;
    let transaction = Transaction::new(/* ... */);

    // Direct Solana RPC call
    let signature = state.solana_client
        .send_and_confirm_transaction(&transaction)?;

    // Now try testing this...
}

#[cfg(test)]
mod tests {
    #[tokio::test]
    async fn test_issue_diploma() {
        // 😱 Test needs real Solana!
        // Options:
        // 1. Spin up solana-test-validator (slow)
        // 2. Use devnet (unstable)
        // 3. Hardcode mocks... everywhere (maintenance nightmare)

        // Result: test is either slow, brittle, or both
    }
}

What's wrong here?

• 🔒 Vendor lock-in: Want to switch to Ethereum? Good luck rewriting everything

• 🐌 Slow tests: Each test waits for real transactions (30-60 seconds)

• 💸 Expensive tests: Devnet requires test SOL, has rate limits

• 🎲 Unstable CI/CD: Network down? All tests red

• 🍝 Spaghetti code: Business logic tangled with blockchain details

The Problem: When Blockchain Becomes an Anchor

While developing our diploma verification service, we hit the classic tight coupling problem. Our code directly depended on the Solana RPC client, creating a whole bouquet of issues:

• Masochistic tests: Every test required a real blockchain connection

• Slow development: Waiting for transaction confirmations on every test run is a special kind of hell

• Brittle CI/CD: Tests failed due to network issues, not actual bugs

• Vendor lock-in: Switching to another blockchain meant rewriting most of the service

Solution Architecture: A Visual Guide

Before diving into code, let's look at the big picture of our architecture:

graph TD
    HTTP[HTTP Requests] --> AxumRouter

    %% =====================
    %% Axum Router
    %% =====================
    subgraph AxumRouter [Axum Router]
        direction TB
        Issue[/issue/]:::router
        Verify[/verify/]:::router
        Creds[/credentials/]:::router
    end

    AxumRouter --> AppState

    %% =====================
    %% AppState
    %% =====================
    subgraph AppState
        direction TB
        ChainClient["chain_client:
Arc(dyn ChainClient)"]
        IssuerKey["issuer_keypair:
Keypair"]
        DBClient["db_client:
Postgrest"]
    end

    AppState -->|implements| ChainClientTrait

    %% =====================
    %% ChainClient trait
    %% =====================
    subgraph ChainClientTrait ["trait ChainClient"]
        direction TB
        WriteHash["async write_hash()
-> Result"]
        FindByHash["async find_by_hash()
-> Result"]
        HealthCheck["async health_check()
-> Result"]
    end

    ChainClientTrait --> SolanaClient
    ChainClientTrait --> MockClient
    ChainClientTrait --> EthereumClient

    %% =====================
    %% Implementations
    %% =====================
    subgraph Production
        SolanaClient["SolanaClient
Real RPC
Real txs
Real fees"]
    end

    subgraph Testing
        MockClient["MockClient
HashMap
Instant
No network"]
    end

    subgraph Future
        EthereumClient["EthereumClient
(easily added)"]
    end

    %% =====================
    %% External systems
    %% =====================
    SolanaClient --> SolanaBlockchain["Solana Blockchain"]
    MockClient --> InMemory["In-memory storage"]

    %% =====================
    %% Styles
    %% =====================
    classDef router fill:#f0f0f0,stroke:#333,stroke-width:1px

Data Flow in Different Environments:

graph BT
    subgraph TESTING
        T1[Test] --> T2[Handler]
        T2 --> T3[AppState]
        T3 --> T4[MockClient]
        T4 --> T5[HashMap in Memory]
        T3 -.-> T6["(dyn ChainClient)"]
    end

    subgraph PRODUCTION
        P1[Request] --> P2[Handler]
        P2 --> P3[AppState]
        P3 --> P4[SolanaClient]
        P4 --> P5[Solana Network]
        P3 -.-> P6["(dyn ChainClient)"]
    end

    style PRODUCTION fill:#e1f5ff
    style TESTING fill:#fff4e1
    style P6 fill:#ffebee
    style T6 fill:#ffebee

The Solution: ChainClient Trait as Our Contract with the Blockchain

Instead of dragging a concrete Solana client implementation everywhere, we created an abstraction - a trait that describes what any blockchain client should be able to do, without specifying how:

// ./internal/blockchain/mod.rs
use async_trait::async_trait;

#[async_trait]
pub trait ChainClient: Send + Sync {
    /// Writes a hash to the blockchain.
    async fn write_hash(&self, hash: &str, meta: &Diploma)
        -> Result<BlockchainRecord, AppError>;

    /// Looks up a hash on the blockchain.
    async fn find_by_hash(&self, hash: &str) 
        -> Result<Option<BlockchainRecord>, AppError>;

    /// Performs a health check on the connection.
    async fn health_check(&self) -> Result<(), AppError>;
}

Why async_trait?

Notice the #[async_trait] macro? This isn't a whim - it's a necessity. At the time of writing, async functions in traits are still not stabilized in Rust (though work is progressing rapidly). The async-trait library solves this elegantly: under the hood, it transforms:

async fn write_hash(&self, ...) -> Result<...>

Into something like:

fn write_hash(&self, ...) -> Box<dyn Future<Output = Result<...>> + Send>

Yes, this adds a small overhead due to dynamic dispatch and heap allocation, but for I/O operations (and blockchain work is always I/O), it's absolutely negligible.

Production Implementation: SolanaChainClient

Now let's look at the concrete implementation for Solana. It encapsulates all the complexity of working with RPC and transactions:

// ./internal/blockchain/solana.rs
pub struct SolanaChainClient {
    rpc_client: RpcClient,
    issuer_keypair: Keypair,
}

impl SolanaChainClient {
    pub fn new(rpc_url: String, issuer_keypair: Keypair) -> Result<Self, AppError> {
        let rpc_client = RpcClient::new(rpc_url);
        Ok(Self {
            rpc_client,
            issuer_keypair,
        })
    }
}

#[async_trait]
impl ChainClient for SolanaChainClient {
    async fn write_hash(
        &self,
        hash: &str,
        _meta: &Diploma,
    ) -> Result<BlockchainRecord, AppError> {
        // Using Memo Program to record the hash
        let memo_program_id = 
            Pubkey::from_str("MemoSq4gqABAXKb96qnH8TysNcWxMyWCqXgDLGmfcHr")?;

        let instruction = Instruction::new_with_bytes(
            memo_program_id,
            hash.as_bytes(),
            vec![],
        );

        // Get latest blockhash and send transaction
        let latest_blockhash = self.rpc_client.get_latest_blockhash()?;
        let message = Message::new(&[instruction], Some(&self.issuer_keypair.pubkey()));
        let mut transaction = Transaction::new_unsigned(message);

        transaction.sign(&[&self.issuer_keypair], latest_blockhash);

        let signature = self.rpc_client
            .send_and_confirm_transaction(&transaction)?;

        Ok(BlockchainRecord {
            tx_id: signature.to_string(),
            block_time: None,
            raw_meta: Some(hash.to_string()),
        })
    }

    // Other methods...
}

All the Solana magic (working with instructions, signatures, blockhashes) is hidden inside the implementation. For the rest of the code, it's just "something that can write hashes to a blockchain."

MockChainClient: A Developer's Testing Paradise

And here's where the real magic begins - our mock implementation for tests:

// ./internal/blockchain/mock.rs
pub struct MockChainClient {
    storage: Mutex<HashMap<String, BlockchainRecord>>,
}

impl MockChainClient {
    pub fn new() -> Self {
        Self {
            storage: Mutex::new(HashMap::new()),
        }
    }
}

#[async_trait]
impl ChainClient for MockChainClient {
    async fn write_hash(
        &self,
        hash: &str,
        _meta: &Diploma,
    ) -> Result<BlockchainRecord, AppError> {
        let mut storage = self.storage.lock().unwrap();
        let record = BlockchainRecord {
            tx_id: format!("mock_tx_{}", hash),
            block_time: Some(Utc::now()),
            raw_meta: Some(hash.to_string()),
        };
        storage.insert(hash.to_string(), record.clone());
        Ok(record)
    }

    async fn find_by_hash(&self, hash: &str) -> Result<Option<BlockchainRecord>, AppError> {
        let storage = self.storage.lock().unwrap();
        Ok(storage.get(hash).cloned())
    }

    async fn health_check(&self) -> Result<(), AppError> {
        Ok(()) // Always healthy!
    }
}

Instead of a real blockchain, we use a simple in-memory HashMap. Transactions are "confirmed" instantly, no network delays, no fees. Now we can write a test for the issue_diploma handler that runs in milliseconds:

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_issue_diploma_creates_blockchain_record() {
        // Arrange: create mock client instead of real one
        let mock_client = Arc::new(MockChainClient::new());
        let app_state = Arc::new(AppState {
            chain_client: mock_client.clone(),
            // ... other fields
        });

        // Act: call business logic
        let result = issue_diploma(State(app_state), test_multipart).await;

        // Assert: verify hash was written
        assert!(result.is_ok());
        let response = result.unwrap();

        // We can even verify the hash was actually saved
        let record = mock_client.find_by_hash(&response.hash).await.unwrap();
        assert!(record.is_some());
        assert_eq!(record.unwrap().tx_id, format!("mock_tx_{}", response.hash));
    }
}

Dependency Injection via AppState

Now for the interesting part - how it all comes together in a real application. We're using Axum (an excellent web framework for Rust), and all the magic happens in AppState:

// ./internal/api/router.rs
pub struct AppState {
    pub chain_client: Arc<dyn ChainClient>,  // <- There it is!
    pub issuer_keypair: Keypair,
    pub db_client: Postgrest,
}

pub async fn create_router() -> Result<Router, AppError> {
    // Load configuration
    let config = Config::from_env()?;

    // Create concrete implementation for production
    let solana_client = SolanaChainClient::new(
        config.solana_rpc_url, 
        client_keypair
    )?;

    // Package into AppState
    let app_state = Arc::new(AppState {
        chain_client: Arc::new(solana_client), // <- Dependency injection!
        issuer_keypair,
        db_client,
    });

    // Create router with injected state
    Ok(Router::new()
        .route("/issue", post(issue_diploma))
        .route("/verify/:hash", get(verify_diploma))
        .with_state(app_state))
}

What's dyn ChainClient?

Notice Arc<dyn ChainClient>? This is a trait object - a way to store any type implementing the ChainClient trait without knowing the concrete type at compile time.

• dyn tells the compiler: "this will be some type implementing ChainClient, but which one exactly - we'll find out at runtime"

• Arc is needed for safe sharing between threads (Axum handles requests in parallel)

dyn Trait vs Generics: Battle of the Titans

You might ask: "Why not use generics?" Excellent question! Let's compare:

Option with Generics (Static Dispatch):

pub struct AppState<C: ChainClient> {
    pub chain_client: Arc<C>,
    // ...
}

Pros:

• Maximum performance (compiler can inline calls)

• No vtable lookup overhead

Cons:

• Monomorphization bloats the binary (separate code copy generated for each type C)

• Can't change implementation at runtime

• All handlers must also become generic functions

Our Choice: Trait Objects (Dynamic Dispatch):

pub struct AppState {
    pub chain_client: Arc<dyn ChainClient>,
    // ...
}

Pros:

• Flexibility: can choose implementation at runtime (e.g., based on environment variable)

• Smaller binary size

• Easier to integrate with web frameworks

Cons:

• Small overhead for calls (vtable lookup)

• Need Arc to work with trait objects

For our use case, the choice is clear: the overhead of vtable lookup is negligible compared to network call times to the blockchain. And the flexibility we gain is priceless.

Conclusion: Architecture That Evolves With You

What we achieved:

Lightning-fast tests: Unit tests run in milliseconds without requiring a real blockchain

Easy blockchain swapping: Want to add Ethereum? Just write EthereumChainClient implementing ChainClient

Clear separation of concerns: Business logic knows nothing about Solana implementation details

Simplified debugging: Can use MockChainClient even in development environment for rapid iteration

This approach isn't a silver bullet, but for services interacting with external systems (whether blockchains, payment systems, or third-party APIs), it provides enormous flexibility and testability.

Remember: good architecture isn't one that anticipates all future changes, but one that makes them easy to implement when needed. Rust traits give us exactly this capability.

P.S. If you're still writing tests that require real connections to external services - give this approach a try. Your colleagues (and your CI/CD pipeline) will thank you!

Introduction: Two Sources of Truth, One Big Headache

Picture this: you're building a diploma verification system. The requirements seem straightforward—data must be immutable (hello, blockchain) while remaining queryable at speed (hello, PostgreSQL). The obvious solution? Write to both. But as every battle-scarred engineer knows, the devil lurks in the implementation details.

Our project uses the dual-write pattern:

• Solana — guarantees immutability and transparency for issued diplomas

• PostgreSQL (Supabase) — enables fast queries and complex analytics

Looks great on architecture diagrams, but production tells a different story. The killer issue? Partial failures. Your Solana transaction succeeds, the diploma is forever etched into the blockchain, but then PostgreSQL decides to take a coffee break. The user gets their confirmation, but half your system has no idea their diploma exists.

Today I'll walk you through how we faced this beast head-on and the patterns we deployed to tame it.

Anatomy of a Failure: Where Things Break

Let's dive into the actual code from our internal/api/handlers.rs. The issue_diploma function is where the magic happens... and where everything can go sideways:

pub async fn issue_diploma(
    State(state): State<Arc<AppState>>,
    mut multipart: Multipart,
) -> Result<Json<IssueResponse>, AppError> {
    // ... multipart data parsing ...

    // Generate diploma hash
    let hash = hashing::generate_hash(
        &file_bytes,
        &req.issuer_id,
        &req.recipient_id,
        issued_at,
        req.serial.as_deref(),
    );

    // Sign the hash with private key
    let signature = hashing::sign_hash(&hash, &state.issuer_keypair)?;

    let diploma = Diploma {
        hash: hash.clone(),
        issuer_id: req.issuer_id.clone(),
        recipient_id: req.recipient_id.clone(),
        signature: Some(signature.clone()),
        issued_at,
        serial: req.serial.clone(),
        ipfs_cid: None,
    };

    // CRITICAL POINT #1: Writing to Solana blockchain
    // This operation can take 1-3 seconds and costs money (gas)
    let chain_record = state.chain_client.write_hash(&hash, &diploma).await?;

    // Prepare data for database
    let credential_data = serde_json::json!({
        "hash": &diploma.hash,
        "issuer_id": &diploma.issuer_id,
        "recipient_id": &diploma.recipient_id,
        "solana_tx_id": &chain_record.tx_id,
        "issued_at": diploma.issued_at.to_rfc3339(),
    });

    // CRITICAL POINT #2: Writing to PostgreSQL
    // And here's where disaster can strike
    let db_response = state
        .db_client
        .from("credentials")
        .insert(credential_data.to_string())
        .execute()
        .await;

    // Handle database error
    let db_response = match db_response {
        Ok(response) => response,
        Err(e) => {
            tracing::error!("Database request failed: {}", e);
            return Err(AppError::Database(format!("Database request failed: {}", e)));
        }
    };

    if !db_response.status().is_success() {
        let status = db_response.status();
        let error_body = db_response.text().await.unwrap_or_default();

        // HERE IT IS - THE CRITICAL INCONSISTENCY!
        tracing::error!(
            "CRITICAL INCONSISTENCY: Failed to save to Supabase after successful Solana transaction. \
             tx_id: {}, hash: {}. Status: {}. Body: {}",
            chain_record.tx_id,
            hash,
            status,
            error_body
        );

        // We've already written to blockchain, can't roll back!
        return Err(AppError::Internal(
            "Failed to save credential record after blockchain confirmation.".to_string(),
        ));
    }

    // If all is well - return success response
    Ok(Json(IssueResponse {
        hash,
        tx_id: chain_record.tx_id,
        signature: Some(signature),
        issued_at,
    }))
}

Here's the sequence of operations and the failure point visualized:

┌──────────┐      ┌────────────┐     ┌─────────┐      ┌────────────┐
│  Client  │────▶│ Rust API   │───▶│ Solana  │────▶│  SUCCESS   │
└──────────┘      └────────────┘     └─────────┘      └────────────┘
                        │                                    │
                        │                                    ▼
                        │                             ┌────────────┐
                        └───────────────────────────▶│ PostgreSQL │
                                                      └────────────┘
                                                            │
                                                            ▼
                                                     ┌────────────┐
                                                     │   FAIL!    │
                                                     └────────────┘
                                                            │
                                                            ▼
                                                ┌──────────────────────┐
                                                │ DATA DRIFT:          │
                                                │ • Blockchain: ✓      │
                                                │ • Database: ✗        │
                                                └──────────────────────┘

The Fallout

What happens after such a failure? Several unpleasant scenarios:

1. Users can't find their diploma through API queries to the database

2. Analytics become impossible — database has incomplete data

3. Audit nightmares — blockchain has the record, reports don't

4. Duplication on retry — users might attempt to issue the diploma again

Theory Meets Practice: Patterns to the Rescue

The Consistency Problem: Choosing a Strategy

In distributed systems, there are two main approaches to consistency:

1. Strong Consistency — all nodes see identical data at the same moment. This is expensive and complex, especially when one node is a public blockchain.

2. Eventual Consistency — data may temporarily differ, but will eventually converge to a consistent state.

We chose eventual consistency. Why? Once a Solana transaction is confirmed, it's irreversible. There's no rollback. So we need to guarantee that PostgreSQL will eventually receive this data.

The Saga Pattern: Long-Running Transactions with Compensation

The Saga pattern breaks a distributed transaction into a sequence of local transactions. Each step can have a compensating transaction for rollback.

Here's how it could look in our case:

// Pseudocode for diploma issuance saga
enum SagaStep {
    SaveToDatabase,      // Step 1
    WriteToBlockchain,   // Step 2
    UpdateDatabaseStatus // Step 3
}

async fn issue_diploma_saga(diploma: Diploma) -> Result<(), SagaError> {
    // Step 1: Save to DB with "pending" status
    let db_record = match save_to_database_with_status(&diploma, "pending").await {
        Ok(record) => record,
        Err(e) => {
            // Nothing to roll back, just exit
            return Err(SagaError::DatabaseFailed(e));
        }
    };

    // Step 2: Write to blockchain
    let tx_id = match write_to_blockchain(&diploma).await {
        Ok(tx) => tx,
        Err(e) => {
            // Compensating transaction: mark record as failed
            mark_database_record_failed(&db_record.id).await?;
            return Err(SagaError::BlockchainFailed(e));
        }
    };

    // Step 3: Update DB status to "confirmed"
    match update_database_status(&db_record.id, "confirmed", &tx_id).await {
        Ok(_) => Ok(()),
        Err(e) => {
            // Compensation is tricky here: blockchain already has the record
            // Can only mark for manual intervention
            mark_for_manual_reconciliation(&db_record.id, &tx_id).await?;
            Err(SagaError::InconsistentState(e))
        }
    }
}

The problem with Saga in blockchain: Compensating transactions in Solana cost money (gas) and don't actually remove previous entries—they add new ones. This makes the pattern expensive and complex.

Idempotency and Retries

Idempotency is the property of an operation yielding the same result on repeated calls. In our context, it's critical.

Here's how we could add a retry mechanism:

use tokio::time::{sleep, Duration};

async fn write_to_database_with_retry(
    db_client: &Postgrest,
    data: serde_json::Value,
    max_retries: u32,
) -> Result<(), AppError> {
    let mut retries = 0;
    let mut backoff = Duration::from_millis(100);

    loop {
        match db_client
            .from("credentials")
            .insert(data.to_string())
            .execute()
            .await 
        {
            Ok(response) if response.status().is_success() => {
                return Ok(());
            }
            Ok(response) if response.status() == 409 => {
                // Conflict - record already exists (idempotency!)
                tracing::info!("Record already exists, considering it success");
                return Ok(());
            }
            Ok(_) | Err(_) if retries < max_retries => {
                retries += 1;
                tracing::warn!(
                    "Database write failed, retry {}/{} after {:?}",
                    retries, max_retries, backoff
                );
                sleep(backoff).await;
                backoff *= 2; // Exponential backoff
            }
            _ => {
                return Err(AppError::Database(
                    "Failed after maximum retries".to_string()
                ));
            }
        }
    }
}

The downside: If the database is down for an extended period (say, scheduled maintenance), the user waits. Meanwhile, the blockchain transaction is already done!

Our Solution: Outbox Pattern with Background Reconciliation

After analyzing various approaches, we settled on combining two patterns:

The Transactional Outbox Pattern

The essence of the Outbox pattern: instead of writing directly to two systems, we make one atomic transaction to the primary storage, including an event in an outbox table.

Here's how our architecture changes:

// New structure for outbox
#[derive(Serialize, Deserialize)]
struct OutboxEvent {
    id: Uuid,
    event_type: String,
    payload: serde_json::Value,
    status: String, // "pending", "processing", "completed", "failed"
    created_at: DateTime<Utc>,
    processed_at: Option<DateTime<Utc>>,
    retry_count: u32,
    error_message: Option<String>,
}

// Modified issue_diploma function
pub async fn issue_diploma_with_outbox(
    State(state): State<Arc<AppState>>,
    mut multipart: Multipart,
) -> Result<Json<IssueResponse>, AppError> {
    // ... parsing and hash generation as before ...

    // IMPORTANT: First write to DB in one transaction
    let mut transaction = state.db_client.begin_transaction().await?;

    // Save diploma with "pending_blockchain" status
    let credential_data = serde_json::json!({
        "hash": &diploma.hash,
        "issuer_id": &diploma.issuer_id,
        "recipient_id": &diploma.recipient_id,
        "status": "pending_blockchain",
        "issued_at": diploma.issued_at.to_rfc3339(),
    });

    transaction
        .from("credentials")
        .insert(credential_data.to_string())
        .execute()
        .await?;

    // Add event to outbox
    let outbox_event = serde_json::json!({
        "id": Uuid::new_v4(),
        "event_type": "WRITE_TO_BLOCKCHAIN",
        "payload": serde_json::to_value(&diploma)?,
        "status": "pending",
        "created_at": Utc::now(),
        "retry_count": 0,
    });

    transaction
        .from("outbox_events")
        .insert(outbox_event.to_string())
        .execute()
        .await?;

    // Commit transaction - all or nothing!
    transaction.commit().await?;

    // Return response to user
    Ok(Json(IssueResponse {
        hash: diploma.hash,
        tx_id: "pending".to_string(), // Will be updated asynchronously
        signature: Some(signature),
        issued_at: diploma.issued_at,
    }))
}

Now we need a background processor for outbox events:

// Background worker for outbox processing
async fn outbox_processor(state: Arc<AppState>) {
    loop {
        // Fetch unprocessed events
        let events = fetch_pending_outbox_events(&state.db_client).await;

        for event in events {
            match event.event_type.as_str() {
                "WRITE_TO_BLOCKCHAIN" => {
                    process_blockchain_write(event, &state).await;
                }
                _ => {
                    tracing::warn!("Unknown event type: {}", event.event_type);
                }
            }
        }

        // Sleep before next iteration
        tokio::time::sleep(Duration::from_secs(5)).await;
    }
}

async fn process_blockchain_write(
    event: OutboxEvent, 
    state: &Arc<AppState>
) {
    let diploma: Diploma = serde_json::from_value(event.payload.clone())
        .expect("Failed to deserialize diploma");

    // Attempt blockchain write
    match state.chain_client.write_hash(&diploma.hash, &diploma).await {
        Ok(chain_record) => {
            // Success! Update statuses
            let mut transaction = state.db_client.begin_transaction().await.unwrap();

            // Update credential
            transaction
                .from("credentials")
                .update(serde_json::json!({
                    "status": "confirmed",
                    "solana_tx_id": chain_record.tx_id,
                }).to_string())
                .eq("hash", &diploma.hash)
                .execute()
                .await
                .unwrap();

            // Mark event as processed
            transaction
                .from("outbox_events")
                .update(serde_json::json!({
                    "status": "completed",
                    "processed_at": Utc::now(),
                }).to_string())
                .eq("id", event.id.to_string())
                .execute()
                .await
                .unwrap();

            transaction.commit().await.unwrap();
        }
        Err(e) => {
            // Error - increment retry counter
            update_outbox_event_retry(&state.db_client, event.id, e.to_string()).await;
        }
    }
}

Reconciliation Job: The Safety Net

Even with the Outbox pattern, things can go wrong. So we added a background reconciliation process:

// Periodic data reconciliation between Solana and PostgreSQL
async fn reconciliation_job(state: Arc<AppState>) {
    loop {
        tracing::info!("Starting reconciliation check...");

        // Fetch recent transactions from Solana (last hour)
        let cutoff_time = Utc::now() - Duration::from_secs(3600);
        let blockchain_records = fetch_recent_blockchain_transactions(
            &state.chain_client,
            cutoff_time
        ).await;

        for record in blockchain_records {
            // Check if record exists in PostgreSQL
            let db_result = state
                .db_client
                .from("credentials")
                .select("hash")
                .eq("hash", &record.hash)
                .single()
                .execute()
                .await;

            if db_result.is_err() || !db_result.unwrap().status().is_success() {
                // Record missing in DB - add it
                tracing::warn!(
                    "Found orphaned blockchain record: hash={}, tx_id={}", 
                    record.hash, 
                    record.tx_id
                );

                // Recover record from blockchain
                let recovery_data = serde_json::json!({
                    "hash": record.hash,
                    "solana_tx_id": record.tx_id,
                    "status": "recovered_from_blockchain",
                    "recovered_at": Utc::now(),
                    // Other fields from transaction metadata
                });

                match state
                    .db_client
                    .from("credentials")
                    .insert(recovery_data.to_string())
                    .execute()
                    .await 
                {
                    Ok(_) => {
                        tracing::info!("Successfully recovered record: {}", record.hash);

                        // Alert the team
                        send_alert(
                            "Data inconsistency detected and fixed",
                            &format!("Recovered hash {} from blockchain", record.hash)
                        ).await;
                    }
                    Err(e) => {
                        tracing::error!("Failed to recover record: {}", e);
                    }
                }
            }
        }

        // Run reconciliation every 5 minutes
        tokio::time::sleep(Duration::from_secs(300)).await;
    }
}

Visualizing the new approach:

┌──────────┐     ┌────────────┐      ┌─────────────┐
│  Client  │───▶│ Rust API   │────▶│ PostgreSQL  │
└──────────┘     └────────────┘      │ + Outbox    │
                                     └─────────────┘
                                            │
                                            ▼
                                     ┌─────────────┐
                                     │   SUCCESS   │
                                     │  (Atomic)   │
                                     └─────────────┘
                                            │
                         ┌──────────────────┼──────────────────┐
                         ▼                  ▼                  ▼
                ┌─────────────────┐ ┌──────────────┐ ┌──────────────┐
                │ Outbox Processor│ │Reconciliation│ │  Monitoring  │
                │   (Async)       │ │     Job      │ │   & Alerts   │
                └─────────────────┘ └──────────────┘ └──────────────┘
                         │                  │
                         ▼                  ▼
                   ┌──────────┐      ┌──────────┐
                   │  Solana  │◀────│  Check   │
                   └──────────┘      └──────────┘

Room for Improvement: Looking Ahead

Queue with Retries

Instead of a simple outbox in the DB, we could use a proper message queue:

// Integration with Redis Streams for more reliable delivery
use redis::AsyncCommands;

async fn publish_to_queue(
    redis_client: &redis::Client,
    diploma: &Diploma,
) -> Result<(), AppError> {
    let mut conn = redis_client.get_async_connection().await?;

    let event = serde_json::json!({
        "type": "WRITE_TO_BLOCKCHAIN",
        "payload": diploma,
        "timestamp": Utc::now().to_rfc3339(),
        "retry_count": 0,
    });

    // Add to Redis Stream with auto-generated ID
    conn.xadd(
        "diploma:outbox",
        "*",
        &[("event", serde_json::to_string(&event)?)],
    ).await?;

    Ok(())
}

// Consumer with group for guaranteed delivery
async fn consume_from_queue(redis_client: &redis::Client, state: Arc<AppState>) {
    let mut conn = redis_client.get_async_connection().await.unwrap();

    // Create consumer group
    let _: Result<(), _> = conn.xgroup_create_mkstream(
        "diploma:outbox",
        "blockchain_writers",
        "$",
    ).await;

    loop {
        // Read events from queue
        let events: Vec<StreamReadReply> = conn.xreadgroup(
            &["diploma:outbox"],
            "blockchain_writers",
            "worker_1",
            &[">"],
            Some(1),
            None,
        ).await.unwrap();

        for event in events {
            // Process and acknowledge
            process_event(event, &state).await;
            conn.xack("diploma:outbox", "blockchain_writers", &[event.id]).await.unwrap();
        }
    }
}

Monitoring and Alerting

Critical to track system state:

use prometheus::{register_counter_vec, register_histogram_vec, CounterVec, HistogramVec};

lazy_static! {
    static ref INCONSISTENCY_COUNTER: CounterVec = register_counter_vec!(
        "diploma_inconsistencies_total",
        "Total number of data inconsistencies detected",
        &["type"]
    ).unwrap();

    static ref RECONCILIATION_DURATION: HistogramVec = register_histogram_vec!(
        "reconciliation_duration_seconds",
        "Time taken to reconcile records",
        &["status"]
    ).unwrap();
}

// Using metrics in code
async fn monitor_inconsistency(inconsistency_type: &str) {
    INCONSISTENCY_COUNTER
        .with_label_values(&[inconsistency_type])
        .inc();

    // Alert if too many inconsistencies
    let total = INCONSISTENCY_COUNTER
        .with_label_values(&[inconsistency_type])
        .get();

    if total > 10.0 {
        send_critical_alert(
            "High inconsistency rate detected",
            &format!("Type: {}, Count: {}", inconsistency_type, total)
        ).await;
    }
}

Event Sourcing for Full Traceability

We could go further and store all events as an immutable log:

#[derive(Serialize, Deserialize)]
enum DiplomaEvent {
    Created {
        hash: String,
        issuer_id: String,
        recipient_id: String,
        timestamp: DateTime<Utc>,
    },
    BlockchainWriteRequested {
        hash: String,
        timestamp: DateTime<Utc>,
    },
    BlockchainWriteCompleted {
        hash: String,
        tx_id: String,
        timestamp: DateTime<Utc>,
    },
    BlockchainWriteFailed {
        hash: String,
        error: String,
        retry_count: u32,
        timestamp: DateTime<Utc>,
    },
    ReconciliationDetected {
        hash: String,
        source: String, // "blockchain" or "database"
        timestamp: DateTime<Utc>,
    },
}

// This gives us complete history for every diploma
async fn append_event(
    db_client: &Postgrest,
    event: DiplomaEvent,
) -> Result<(), AppError> {
    let event_data = serde_json::json!({
        "event_type": event.variant_name(),
        "payload": serde_json::to_value(&event)?,
        "timestamp": Utc::now(),
    });

    db_client
        .from("diploma_events")
        .insert(event_data.to_string())
        .execute()
        .await?;

    Ok(())
}

Conclusion: Lessons from the Trenches

Working with dual-writes between Solana and PostgreSQL taught us several hard lessons:

1. Never trust sequential calls — just because the first one succeeds doesn't guarantee the second will. Especially when the first is an irreversible blockchain operation.

2. Design for failure — it's not a question of if the system will fail, but when. The Outbox pattern and background reconciliation aren't redundancy; they're necessities.

3. Eventual consistency is your friend — don't try to achieve strong consistency between blockchain and traditional databases. It's expensive, complex, and often impossible.

4. Monitoring is critical — better to get an alert about drift within a minute than hear about it from a user a week later.

5. Idempotency saves lives — design operations so they can be safely retried. This simplifies recovery from failures.

For fellow engineers working with Web3 backends in Rust: blockchain isn't a silver bullet. It's a powerful tool, but it requires careful system design. Dual-writes seem simple until you hit your first production failure at 3 AM.

Remember: in distributed systems, everything that can go wrong will go wrong. Design accordingly.

Useful Links

• Saga Pattern - Microservices.io

• Transactional Outbox Pattern

• Event Sourcing in Rust

• Building on Solana with Rust

• PostgreSQL and Distributed Systems

• Designing Data-Intensive Applications — the bible for distributed systems

If you have experience solving similar problems or questions about implementation, let's discuss in the comments. I'm particularly interested in hearing about alternative approaches to syncing blockchain with traditional databases.