While Python (scikit-learn / fuzzywuzzy) is the go-to for this, it can be slow at scale and requires heavy dependencies.

In this post, I’ll show you how to build a blazing fast, production-grade Standardization Engine using Rust for the backend and React for the UI. We will go beyond simple regex and implement:

1. Real-time Master Data: Fetching live tickers from the SEC (US Securities and Exchange Commission).

2. Fuzzy Logic: Using the Jaro-Winkler algorithm to handle typos ("Amzn" → "Amazon").

3. Modern UI: A polished React frontend with Dark Mode and CSV Export.

🏗️ The Architecture

• Backend: Rust (Actix-web)

  • Speed: Handles thousands of requests per second.

  • Logic: strsim crate for string distance calculations.

  • Data: Fetches company_tickers.json from SEC.gov on startup.

• Frontend: React (Vite)

  • UX: Real-time lookup, Batch CSV processing, Dark/Light mode toggle.

🚀 Part 1: The Rust Backend

We use Actix-web for the server and strsim for the fuzzy matching logic.

1. Initialize the Project

mkdir company-standardizer
cd company-standardizer
cargo new backend
cd backend

2. Dependencies (Cargo.toml)

We need reqwest to fetch the SEC data and strsim for the math.

[package]
name = "backend"
version = "0.1.0"
edition = "2021"

[dependencies]
actix-web = "4"
actix-cors = "0.6"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
strsim = "0.11" # The magic fuzzy matching crate
reqwest = { version = "0.11", features = ["json", "blocking"] }
tokio = { version = "1", features = ["full"] }

3. The Logic (src/main.rs)

This is the core engine. It employs a "Waterfall" matching strategy:

1. Ticker Map: O(1) Lookup (e.g., "MSFT" -> "Microsoft").

2. Jaro-Winkler: Measures string similarity. If the score > 0.75, we consider it a match.

use actix_cors::Cors;
use actix_web::{web, App, HttpResponse, HttpServer, Responder};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::Mutex;
use strsim::jaro_winkler;

// --- Data Structures ---
#[derive(Clone, Serialize, Deserialize, Debug)]
struct CompanyReference {
    title: String,
    ticker: String,
}

#[derive(Deserialize, Debug)]
struct SecCompany {
    ticker: String,
    title: String,
}

#[derive(Serialize, Debug)]
struct MatchResult {
    input: String,
    standardized_name: Option<String>,
    ticker: Option<String>,
    score: f64,
    method: String,
}

#[derive(Deserialize)]
struct LookupRequest { name: String }

#[derive(Deserialize)]
struct BatchRequest { names: Vec<String> }

// --- The Engine ---
struct StandardizationEngine {
    master_list: Vec<CompanyReference>,
    ticker_map: HashMap<String, String>,
}

impl StandardizationEngine {
    fn new(companies: Vec<CompanyReference>) -> Self {
        let mut ticker_map = HashMap::new();
        for company in &companies {
            ticker_map.insert(company.ticker.clone(), company.title.clone());
        }
        StandardizationEngine { master_list: companies, ticker_map }
    }

    fn search(&self, query: &str) -> MatchResult {
        let clean_query = query.trim();
        let query_upper = clean_query.to_uppercase();

        // 1. Ticker Lookup (Exact & Fast)
        if let Some(name) = self.ticker_map.get(&query_upper) {
            return MatchResult {
                input: query.to_string(),
                standardized_name: Some(name.clone()),
                ticker: Some(query_upper),
                score: 1.0,
                method: "Ticker Map".to_string(),
            };
        }

        // 2. Fuzzy Search (Jaro-Winkler)
        let mut best_match: Option<CompanyReference> = None;
        let mut best_score = 0.0;

        for company in &self.master_list {
            let score = jaro_winkler(&query_upper, &company.title.to_uppercase());
            if score > best_score {
                best_score = score;
                best_match = Some(company.clone());
            }
        }

        let method = if best_score == 1.0 { "Exact Match" } else { "Jaro-Winkler Fuzzy" };

        // Threshold > 0.75 usually indicates a good match for company names
        if best_score > 0.75 {
            let m = best_match.unwrap();
            MatchResult {
                input: query.to_string(),
                standardized_name: Some(m.title),
                ticker: Some(m.ticker),
                score: best_score,
                method: method.to_string(),
            }
        } else {
            MatchResult {
                input: query.to_string(),
                standardized_name: None,
                ticker: None,
                score: best_score,
                method: "No Match".to_string(),
            }
        }
    }
}

struct AppState {
    engine: Mutex<StandardizationEngine>,
}

// --- API Handlers ---
async fn standardize_one(data: web::Data<AppState>, req: web::Json<LookupRequest>) -> impl Responder {
    let engine = data.engine.lock().unwrap();
    let result = engine.search(&req.name);
    HttpResponse::Ok().json(result)
}

async fn standardize_batch(data: web::Data<AppState>, req: web::Json<BatchRequest>) -> impl Responder {
    let engine = data.engine.lock().unwrap();
    let results: Vec<MatchResult> = req.names.iter().map(|name| engine.search(name)).collect();
    HttpResponse::Ok().json(results)
}

// --- Data Loader ---
async fn fetch_sec_data() -> Vec<CompanyReference> {
    println!("Fetching live data from SEC.gov...");
    let client = reqwest::Client::new();
    // SEC requires a user-agent
    let resp = client.get("https://www.sec.gov/files/company_tickers.json")
        .header("User-Agent", "TestApp (test@example.com)") 
        .send().await;

    if let Ok(response) = resp {
        if let Ok(map) = response.json::<HashMap<String, SecCompany>>().await {
            let mut companies = Vec::new();
            for (_, val) in map {
                companies.push(CompanyReference { title: val.title, ticker: val.ticker });
            }
            println!("Loaded {} companies.", companies.len());
            return companies;
        }
    }
    // Fallback data if API fails
    vec![CompanyReference { title: "Apple Inc.".into(), ticker: "AAPL".into() }]
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let companies = fetch_sec_data().await;
    let engine = StandardizationEngine::new(companies);
    let app_state = web::Data::new(AppState { engine: Mutex::new(engine) });

    println!("Server running at http://127.0.0.1:8080");

    HttpServer::new(move || {
        App::new()
            .wrap(Cors::permissive())
            .app_data(app_state.clone())
            .route("/api/standardize", web::post().to(standardize_one))
            .route("/api/batch-standardize", web::post().to(standardize_batch))
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}

🎨 Part 2: The React Frontend

We use Vite for a fast setup. The frontend features a beautiful card-based layout, a Dark Mode toggle, and CSV export functionality.

1. Setup

cd .. # go to the root directory
npm create vite@latest frontend -- --template react
cd frontend
npm install axios

2. Styling (App.css)

This CSS uses variables for instant theme switching.

:root {
  /* Light Theme Variables */
  --bg-color: #f8f9fa;
  --card-bg: #ffffff;
  --text-main: #2c3e50;
  --text-secondary: #5f6368;
  --accent-color: #2563eb;
  --accent-hover: #1d4ed8;
  --border-color: #e2e8f0;
  --input-bg: #ffffff;
  --table-header: #f1f5f9;
  --table-hover: #f8fafc;
  --shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06);
}

[data-theme='dark'] {
  /* Dark Theme Variables */
  --bg-color: #0f172a;
  --card-bg: #1e293b;
  --text-main: #f1f5f9;
  --text-secondary: #94a3b8;
  --accent-color: #3b82f6;
  --accent-hover: #60a5fa;
  --border-color: #334155;
  --input-bg: #0f172a;
  --table-header: #334155;
  --table-hover: #1e293b; /* Slightly lighter than card bg for hover effect if needed */
  --shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.5);
}

body {
  margin: 0;
  background-color: var(--bg-color);
  color: var(--text-main);
  font-family: 'Inter', system-ui, -apple-system, sans-serif;
  transition: background-color 0.3s, color 0.3s;
  min-height: 100vh;
  display: flex;
  justify-content: center;
}

#root {
  width: 100%;
  max-width: 1000px;
  padding: 2rem;
  text-align: center;
}

/* Header & Toggle */
header {
  display: flex;
  justify-content: space-between;
  align-items: center;
  margin-bottom: 2rem;
  padding-bottom: 1rem;
  border-bottom: 1px solid var(--border-color);
}

h1 {
  font-size: 2rem;
  margin: 0;
  background: linear-gradient(90deg, var(--accent-color), #8b5cf6);
  -webkit-background-clip: text;
  -webkit-text-fill-color: transparent;
}

.theme-toggle {
  background: none;
  border: 1px solid var(--border-color);
  color: var(--text-main);
  font-size: 1.2rem;
  padding: 0.5rem;
  border-radius: 50%;
  cursor: pointer;
  transition: all 0.2s;
  width: 40px;
  height: 40px;
  display: flex;
  align-items: center;
  justify-content: center;
}
.theme-toggle:hover {
  background-color: var(--border-color);
}

/* Cards */
.card {
  background-color: var(--card-bg);
  padding: 2rem;
  border-radius: 12px;
  box-shadow: var(--shadow);
  border: 1px solid var(--border-color);
  margin-bottom: 2rem;
  transition: background-color 0.3s;
}

h2 {
  margin-top: 0;
  color: var(--text-main);
  font-size: 1.25rem;
  margin-bottom: 1.5rem;
  text-align: left;
}

/* Inputs & Buttons */
.search-box {
  display: flex;
  gap: 12px;
}

input[type="text"] {
  flex: 1;
  padding: 12px 16px;
  border-radius: 8px;
  border: 1px solid var(--border-color);
  background-color: var(--input-bg);
  color: var(--text-main);
  font-size: 1rem;
  outline: none;
  transition: border-color 0.2s;
}

input[type="text"]:focus {
  border-color: var(--accent-color);
  box-shadow: 0 0 0 2px rgba(37, 99, 235, 0.2);
}

button.primary-btn {
  background-color: var(--accent-color);
  color: white;
  border: none;
  padding: 12px 24px;
  border-radius: 8px;
  font-weight: 600;
  cursor: pointer;
  transition: background-color 0.2s;
}

button.primary-btn:hover {
  background-color: var(--accent-hover);
}

button:disabled {
  opacity: 0.7;
  cursor: not-allowed;
}

/* File Upload */
.file-upload-label {
  display: inline-block;
  padding: 12px 24px;
  border: 2px dashed var(--border-color);
  border-radius: 8px;
  cursor: pointer;
  color: var(--text-secondary);
  font-weight: 500;
  width: 100%;
  box-sizing: border-box;
  transition: all 0.2s;
}

.file-upload-label:hover {
  border-color: var(--accent-color);
  color: var(--accent-color);
  background-color: rgba(37, 99, 235, 0.05);
}

/* Results Grid (Single) */
.result-grid {
  display: grid;
  grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
  gap: 1.5rem;
  margin-top: 1.5rem;
  text-align: left;
  padding-top: 1.5rem;
  border-top: 1px solid var(--border-color);
}

.result-item label {
  display: block;
  font-size: 0.85rem;
  color: var(--text-secondary);
  margin-bottom: 0.25rem;
  text-transform: uppercase;
  letter-spacing: 0.05em;
}

.result-item .value {
  font-size: 1.1rem;
  font-weight: 600;
  color: var(--text-main);
}

.ticker-badge {
  background-color: var(--border-color);
  color: var(--text-main);
  padding: 2px 6px;
  border-radius: 4px;
  font-size: 0.8rem;
  margin-left: 8px;
  vertical-align: middle;
}

/* Table */
.table-container {
  overflow-x: auto;
  border-radius: 8px;
  border: 1px solid var(--border-color);
}

table {
  width: 100%;
  border-collapse: collapse;
}

th {
  background-color: var(--table-header);
  color: var(--text-secondary);
  font-weight: 600;
  padding: 12px 16px;
  text-align: left;
  font-size: 0.9rem;
}

td {
  padding: 12px 16px;
  border-top: 1px solid var(--border-color);
  color: var(--text-main);
  text-align: left;
}

tr:hover td {
  background-color: var(--table-hover); /* Fixed hover visibility */
}

/* Utilities */
.status-badge {
  padding: 4px 8px;
  border-radius: 4px;
  font-size: 0.85rem;
  font-weight: 600;
}

3. The Logic (App.jsx)

Features:

• Batch Processing: Upload a .txt file, get a table of results.

• CSV Download: Automatically generates a report of the standardization.

• Theme Toggle: Persists your choice in localStorage.

import { useState, useEffect } from 'react'
import axios from 'axios'
import './App.css'

function App() {
    const [inputText, setInputText] = useState('')
    const [result, setResult] = useState(null)
    const [batchResults, setBatchResults] = useState([])
    const [loading, setLoading] = useState(false)
    const [theme, setTheme] = useState('light')

    // Theme Init
    useEffect(() => {
        const savedTheme = localStorage.getItem('theme')
        const prefersDark = window.matchMedia('(prefers-color-scheme: dark)').matches

        if (savedTheme) {
            setTheme(savedTheme)
            document.documentElement.setAttribute('data-theme', savedTheme)
        } else if (prefersDark) {
            setTheme('dark')
            document.documentElement.setAttribute('data-theme', 'dark')
        }
    }, [])

    const toggleTheme = () => {
        const newTheme = theme === 'light' ? 'dark' : 'light'
        setTheme(newTheme)
        document.documentElement.setAttribute('data-theme', newTheme)
        localStorage.setItem('theme', newTheme)
    }

    // --- API Handlers ---

    const handleCheck = async () => {
        if (!inputText) return
        setLoading(true)
        try {
            const res = await axios.post('http://127.0.0.1:8080/api/standardize', { name: inputText })
            setResult(res.data)
        } catch (err) {
            console.error(err)
            alert("Error connecting to backend")
        }
        setLoading(false)
    }

    const handleFileUpload = (e) => {
        const file = e.target.files[0]
        if (!file) return

        const reader = new FileReader()
        reader.onload = async (event) => {
            const text = event.target.result
            const names = text.split('\n').map(n => n.trim()).filter(n => n)

            setLoading(true)
            try {
                const res = await axios.post('http://127.0.0.1:8080/api/batch-standardize', { names })
                setBatchResults(res.data)
            } catch (err) {
                console.error(err)
                alert("Error processing batch")
            }
            setLoading(false)
        }
        reader.readAsText(file)
    }

    // --- CSV Download Logic ---

    const downloadCSV = () => {
        if (batchResults.length === 0) return

        // 1. Define Headers
        const headers = ["Input Name", "Standardized Name", "Ticker", "Method", "Confidence Score"]

        // 2. Format Data Rows
        const csvRows = batchResults.map(item => {
            // Escape quotes in data to prevent CSV breakage
            const safeInput = `"${item.input.replace(/"/g, '""')}"`
            const safeName = `"${(item.standardized_name || "").replace(/"/g, '""')}"`
            const safeTicker = `"${(item.ticker || "").replace(/"/g, '""')}"`
            const safeMethod = `"${item.method}"`
            const safeScore = `"${(item.score * 100).toFixed(1)}%"`

            return [safeInput, safeName, safeTicker, safeMethod, safeScore].join(",")
        })

        // 3. Combine and Trigger Download
        const csvContent = [headers.join(","), ...csvRows].join("\n")
        const blob = new Blob([csvContent], { type: 'text/csv;charset=utf-8;' })
        const url = URL.createObjectURL(blob)

        const link = document.createElement("a")
        link.href = url
        link.setAttribute("download", "standardized_companies_report.csv")
        document.body.appendChild(link)
        link.click()
        document.body.removeChild(link)
    }

    // --- Render Helpers ---

    const getScoreColor = (score) => {
        if (score === 1.0) return '#22c55e'
        if (score > 0.85) return '#3b82f6'
        if (score > 0.75) return '#f59e0b'
        return '#ef4444'
    }

    return (
        <div className="app-container">
            <header>
                <div>
                    <h1>Company Standardizer</h1>
                    <span style={{ color: 'var(--text-secondary)', fontSize: '0.9rem' }}>
            Powered by Rust & SEC API
          </span>
                </div>
                <button className="theme-toggle" onClick={toggleTheme} title="Toggle Theme">
                    {theme === 'light' ? '🌙' : '☀️'}
                </button>
            </header>

            {/* Single Lookup Card */}
            <section className="card">
                <h2>Single Entity Lookup</h2>
                <div className="search-box">
                    <input
                        type="text"
                        value={inputText}
                        onChange={(e) => setInputText(e.target.value)}
                        placeholder="Enter Company Name or Ticker (e.g., Apple, NVDA)"
                        onKeyDown={(e) => e.key === 'Enter' && handleCheck()}
                    />
                    <button className="primary-btn" onClick={handleCheck} disabled={loading}>
                        {loading ? 'Searching...' : 'Search'}
                    </button>
                </div>

                {result && (
                    <div className="result-grid">
                        <div className="result-item">
                            <label>Input</label>
                            <div className="value">{result.input}</div>
                        </div>

                        <div className="result-item">
                            <label>Standardized Name</label>
                            <div className="value" style={{ display: 'flex', flexDirection: 'column', alignItems: 'flex-start', gap: '5px' }}>
                                {result.standardized_name ? (
                                    <>
                                        <span>{result.standardized_name}</span>
                                        {result.ticker && (
                                            <span className="ticker-badge" title="Stock Ticker">
                        {result.ticker}
                      </span>
                                        )}
                                    </>
                                ) : (
                                    <span style={{color: 'var(--text-secondary)'}}>No Match</span>
                                )}
                            </div>
                        </div>

                        <div className="result-item">
                            <label>Method</label>
                            <div className="value" style={{ fontSize: '0.95rem' }}>{result.method}</div>
                        </div>

                        <div className="result-item">
                            <label>Confidence</label>
                            <div className="value" style={{ color: getScoreColor(result.score) }}>
                                {(result.score * 100).toFixed(1)}%
                            </div>
                        </div>
                    </div>
                )}
            </section>

            {/* Batch Processing Card */}
            <section className="card">
                <div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center', marginBottom: '1.5rem' }}>
                    <h2 style={{ marginBottom: 0 }}>Batch Processing</h2>

                    {/* Download Button - Only shows when there are results */}
                    {batchResults.length > 0 && (
                        <button
                            onClick={downloadCSV}
                            style={{
                                backgroundColor: '#22c55e', // Green for Excel/CSV
                                color: 'white',
                                border: 'none',
                                padding: '8px 16px',
                                borderRadius: '6px',
                                cursor: 'pointer',
                                fontSize: '0.9rem',
                                fontWeight: '600',
                                boxShadow: '0 2px 4px rgba(0,0,0,0.1)'
                            }}
                        >
                            ⬇ Download CSV
                        </button>
                    )}
                </div>

                <div style={{ marginBottom: '1.5rem' }}>
                    <label className="file-upload-label">
                        <input
                            type="file"
                            onChange={handleFileUpload}
                            accept=".txt,.csv"
                            style={{ display: 'none' }}
                        />
                        {loading ? "Processing..." : "Click to Upload .txt or .csv List"}
                    </label>
                </div>

                {loading && batchResults.length === 0 && <p style={{color: 'var(--text-secondary)'}}>Processing batch file...</p>}

                {batchResults.length > 0 && (
                    <div className="table-container">
                        <table>
                            <thead>
                            <tr>
                                <th>Input</th>
                                <th>Standardized Name</th>
                                <th>Ticker</th>
                                <th>Method</th>
                                <th>Score</th>
                            </tr>
                            </thead>
                            <tbody>
                            {batchResults.map((item, idx) => (
                                <tr key={idx}>
                                    <td>{item.input}</td>
                                    <td>{item.standardized_name || "-"}</td>
                                    <td>
                                        {item.ticker ? <span className="ticker-badge">{item.ticker}</span> : <span style={{color:'var(--text-secondary)'}}>-</span>}
                                    </td>
                                    <td>{item.method}</td>
                                    <td>
                      <span
                          className="status-badge"
                          style={{
                              backgroundColor: `${getScoreColor(item.score)}20`,
                              color: getScoreColor(item.score)
                          }}
                      >
                        {(item.score * 100).toFixed(0)}%
                      </span>
                                    </td>
                                </tr>
                            ))}
                            </tbody>
                        </table>
                    </div>
                )}
            </section>
        </div>
    )
}

export default App

⚡ How to Run

Follow these exact steps to run the application locally.

1. Start the Backend

Open a terminal in the backend/ directory:

# This compiles the Rust code and starts the server
cargo run

Wait until you see: Successfully loaded X companies... Server running at http://127.0.0.1:8080.

2. Start the Frontend

Open a new terminal window in the frontend/ directory:

# Install dependencies (only needed the first time)
npm install

# Start the dev server
npm run dev

Click the URL displayed (usually http://localhost:5173).

3. Test It Out

• Exact Match: Type MSFT. The system detects the ticker and returns Microsoft Corporation.

• Fuzzy Match: Type Amzn. The system uses Jaro-Winkler logic to match it to Amazon.com Inc. with high confidence.

• Batch: Create a text file named test.txt with the content:

    Appl
    Teslaa
    JPMvc
  

  Upload it, view the table, and click Download CSV to get your report.

Conclusion

By combining the performance of Rust with the interactivity of React, we've built a tool that solves a real Data Engineering problem. This architecture is easily extensible, you could swap the SEC API for an internal SQL database or add machine learning models for even smarter matching.

The breakthrough happened when I stopped trying to learn "Rust" and started trying to build "Data Engineering tools" with Rust.

In this post, we will build a classic Extract-Load (EL) pipeline. Instead of Python, Requests, and Pandas, we will use Rust, Reqwest, and SQLite. We will fetch real-time Bitcoin data from a public API, validate the schema strictly (something Python struggles with), and archive it into a database.

Why Rust for Data Engineering?

• Schema Safety: If the API changes its data format, our code fails before processing, not after filling the database with garbage.

• Single Binary: No virtual environments or pip install. The final output is a single file you can drop onto a server or Lambda function.

• Performance: It handles high-throughput async tasks with a fraction of the memory Python requires.

The Stack

We are using the "Big Four" crates (libraries) for Rust data ops:

1. reqwest: The HTTP client (Rust's requests).

2. tokio: The async runtime (handles concurrency).

3. serde: Serialization/Deserialization (turns JSON into Structs).

4. rusqlite: Lightweight SQLite interaction.

Step 1: Project Setup

First, initialize a new project and add the dependencies.

# Create the project
cargo new crypto_ingestor
cd crypto_ingestor

# Add dependencies via CLI (modern Cargo)
cargo add reqwest --features json
cargo add tokio --features full
cargo add rusqlite --features bundled
cargo add serde --features derive

Step 2: Define the Data Contract

In Python, we might just grab data['bitcoin']['usd']. In Rust, we define the structure upfront. This is our contract. If the API violates this, the pipeline stops immediately.

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct CoinGeckoResponse {
    bitcoin: PriceData,
}

#[derive(Debug, Deserialize)]
struct PriceData {
    usd: f64,
    usd_market_cap: f64,
}

Step 3: The "Gotcha" (Handling Real World APIs)

Public APIs often block generic HTTP clients. During development, I ran into a 403 Forbidden or Schema Error because the API rejected the default User-Agent.

Here is the robust, production-ready fetch function. Note how we handle errors: instead of a generic traceback, we log the raw response if parsing fails (acting like a mini Dead Letter Queue).

use std::error::Error;

async fn fetch_price() -> std::result::Result<CoinGeckoResponse, Box<dyn Error>> {
    let url = "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd&include_market_cap=true";

    // 1. Build a client to set headers (mimic a browser)
    let client = reqwest::Client::new();

    let response = client
        .get(url)
        .header("User-Agent", "MyCryptoIngestor/1.0") 
        .send()
        .await?;

    // 2. Check Status Code before parsing
    if !response.status().is_success() {
        return Err(Box::from(format!("API Request failed with status: {}", response.status())));
    }

    // 3. Get raw text to debug schema failures
    let text_content = response.text().await?;

    // 4. Parse JSON
    let parsed_data: CoinGeckoResponse = match serde_json::from_str(&text_content) {
        Ok(data) => data,
        Err(e) => {
            eprintln!("⚠️ RAW RESPONSE RECEIVED: {}", text_content);
            return Err(Box::new(e));
        }
    };

    Ok(parsed_data)
}

Step 4: Database Operations & Verification

We need to create the table if it doesn't exist, insert the data, and then—to prove it worked—read it back within the same app.

use rusqlite::{params, Connection, Result};

fn init_db() -> Result<Connection> {
    let conn = Connection::open("prices.db")?;
    conn.execute(
        "CREATE TABLE IF NOT EXISTS bitcoin_prices (
            id INTEGER PRIMARY KEY,
            price_usd REAL,
            market_cap REAL,
            timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
        )",
        [],
    )?;
    Ok(conn)
}

fn read_history(conn: &Connection) -> Result<()> {
    let mut stmt = conn.prepare("SELECT id, price_usd, timestamp FROM bitcoin_prices ORDER BY id DESC LIMIT 5")?;

    // Map rows to a tuple
    let rows = stmt.query_map([], |row| {
        Ok((
            row.get::<_, i32>(0)?,    // id
            row.get::<_, f64>(1)?,    // price
            row.get::<_, String>(2)?  // timestamp
        ))
    })?;

    println!("\n📊 Recent History (Last 5 records):");
    println!("-------------------------------------");

    for row_result in rows {
        let (id, price, time) = row_result?;
        println!("[{}] {} | Price: ${:.2}", id, time, price);
    }

    Ok(())
}

Step 5: The Full Pipeline (Putting it together)

Here is the complete src/main.rs file.

use rusqlite::{params, Connection, Result};
use serde::Deserialize;
use std::error::Error;

// --- Data Models ---
#[derive(Debug, Deserialize)]
struct CoinGeckoResponse {
    bitcoin: PriceData,
}

#[derive(Debug, Deserialize)]
struct PriceData {
    usd: f64,
    usd_market_cap: f64,
}

// --- Database Functions ---
fn init_db() -> Result<Connection> {
    let conn = Connection::open("prices.db")?;
    conn.execute(
        "CREATE TABLE IF NOT EXISTS bitcoin_prices (
            id INTEGER PRIMARY KEY,
            price_usd REAL,
            market_cap REAL,
            timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
        )",
        [],
    )?;
    Ok(conn)
}

fn read_history(conn: &Connection) -> Result<()> {
    let mut stmt = conn.prepare("SELECT id, price_usd, timestamp FROM bitcoin_prices ORDER BY id DESC LIMIT 5")?;
    let rows = stmt.query_map([], |row| {
        Ok((
            row.get::<_, i32>(0)?,
            row.get::<_, f64>(1)?,
            row.get::<_, String>(2)?
        ))
    })?;

    println!("\n📊 Recent History (Last 5 records):");
    println!("-------------------------------------");
    for row_result in rows {
        let (id, price, time) = row_result?;
        println!("[{}] {} | Price: ${:.2}", id, time, price);
    }
    Ok(())
}

// --- Extract Function ---
async fn fetch_price() -> std::result::Result<CoinGeckoResponse, Box<dyn Error>> {
    let url = "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd&include_market_cap=true";
    let client = reqwest::Client::new();

    let response = client
        .get(url)
        .header("User-Agent", "MyCryptoIngestor/1.0") 
        .send()
        .await?;

    if !response.status().is_success() {
        return Err(Box::from(format!("API Request failed with status: {}", response.status())));
    }

    let text_content = response.text().await?;
    let parsed_data: CoinGeckoResponse = match serde_json::from_str(&text_content) {
        Ok(data) => data,
        Err(e) => {
            eprintln!("⚠️ RAW RESPONSE RECEIVED: {}", text_content);
            return Err(Box::new(e));
        }
    };
    Ok(parsed_data)
}

// --- Main Orchestration ---
#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn Error>> {
    println!("🚀 Starting Ingestion Pipeline...");

    // 1. Initialize DB
    let conn = init_db()?;

    // 2. Fetch Data (Extract)
    println!("📥 Fetching data from API...");
    match fetch_price().await {
        Ok(data) => {
            let price = data.bitcoin.usd;
            let cap = data.bitcoin.usd_market_cap;
            println!("✅ Data received: BTC at ${}", price);

            // 3. Insert into DB (Load)
            conn.execute(
                "INSERT INTO bitcoin_prices (price_usd, market_cap) VALUES (?1, ?2)",
                params![price, cap],
            )?;
            println!("💾 Saved to SQLite database.");

            // 4. Verification
            read_history(&conn)?;
        }
        Err(e) => {
            eprintln!("❌ Pipeline Failed: {}", e);
        }
    }

    Ok(())
}

Running the Pipeline

To run the application:

cargo run

Expected Output:

🚀 Starting Ingestion Pipeline...
📥 Fetching data from API...
✅ Data received: BTC at $90610
💾 Saved to SQLite database.

📊 Recent History (Last 5 records):
-------------------------------------
[1] 2024-01-10 12:30:01 | Price: $90610.00

Conclusion

We just built a fully functional data app in Rust. It handles network failures, parses JSON into strong types, and persists data to disk.

For a Data Engineer, the biggest shift is moving from "Scripting" (where you run code and hope it works) to "Systems Programming" (where you handle every possible error state upfront). It’s stricter, but the result is a pipeline that sleeps soundly at night.

As a Data Engineer, I decided to stop following generic tutorials and build something I actually understand: an ETL pipeline.

In this post, I’ll walk you through building "Rusty Pipe", a command-line tool. We will start by building a strict, type-safe version, and then refactor it into a generic tool that can handle any CSV file.

Step 1: The Setup

First, we use Cargo (Rust's package manager) to create the project.

cargo new rusty_pipe
cd rusty_pipe

We need a few "crates" (libraries). Add these to your Cargo.toml:

[dependencies]
clap = { version = "4.5", features = ["derive"] } # CLI Arguments
serde = { version = "1.0", features = ["derive"] } # Serialization
serde_json = "1.0"
csv = "1.3"

Step 2: Create Dummy Data

Before we code, let's create the data we want to process. Create a file named products.csv in your project root:

id,name,category,price
1,Apple,Fruit,1.20
2,Laptop,Electronics,999.99
3,Banana,Fruit,0.50
4,TV,Electronics,500.00

Step 3: Version 1 - The "Strict" Approach (Type Safety)

In Python/Pandas, types are often inferred. In Rust, we usually define the shape of our data upfront using a Struct. This acts as a contract—if the CSV has bad data (like text in a price column), the program warns us immediately.

Here is the code for src/main.rs. It reads the CSV, filters out cheap items (price < $1.00), and writes to JSON.

use clap::Parser;
use serde::{Deserialize, Serialize};
use std::error::Error;
use std::fs;

// 1. Define CLI Arguments
#[derive(Parser)]
struct Cli {
    input: String,
    output: String,
}

// 2. Define the Schema (The Contract)
#[derive(Debug, Serialize, Deserialize)]
struct Product {
    id: u32,
    name: String,
    category: String,
    price: f64,
}

fn main() -> Result<(), Box<dyn Error>> {
    let args = Cli::parse();
    let mut rdr = csv::Reader::from_path(args.input)?;

    let mut clean_data: Vec<Product> = Vec::new();

    // 3. Stream the data (Memory Efficient!)
    for result in rdr.deserialize() {
        let record: Product = match result {
            Ok(rec) => rec,
            Err(e) => {
                eprintln!("Skipping bad row: {}", e);
                continue;
            }
        };

        // 4. Business Logic: Filter cheap products
        if record.price > 1.0 {
            clean_data.push(record);
        }
    }

    // 5. Write to JSON
    let json_output = serde_json::to_string_pretty(&clean_data)?;
    fs::write(args.output, json_output)?;

    println!("Success! Processed {} records.", clean_data.len());
    Ok(())
}

Running Version 1

Run this in your terminal:

cargo run -- products.csv output.json

If you check output.json, you will see it correctly filtered out the "Banana" (which was $0.50):

[
  {
    "id": 1,
    "name": "Apple",
    "category": "Fruit",
    "price": 1.2
  },
  {
    "id": 2,
    "name": "Laptop",
    "category": "Electronics",
    "price": 999.99
  },
  {
    "id": 4,
    "name": "TV",
    "category": "Electronics",
    "price": 500.0
  }
]

Step 4: The Refactor - Making it Generic

The strict version is great for production pipelines where the schema is known. But what if I want to use this tool on any CSV file, regardless of columns?

We can swap the Struct for a HashMap. This is closer to how Python works, dynamic and flexible.

Changes required:

1. Remove struct Product.

2. Import std::collections::HashMap.

3. Change Vec<Product> to Vec<HashMap<String, String>>.

Here is the updated src/main.rs:

use clap::Parser;
use std::error::Error;
use std::fs;
use std::collections::HashMap; // Import HashMap

#[derive(Parser)]
struct Cli {
    input: String,
    output: String,
}

// NOTE: We removed the Product struct!

fn main() -> Result<(), Box<dyn Error>> {
    let args = Cli::parse();
    let mut rdr = csv::Reader::from_path(args.input)?;

    // Change the Vector to store Maps (Key=String, Value=String)
    let mut clean_data: Vec<HashMap<String, String>> = Vec::new();

    for result in rdr.deserialize() {
        // Rust automatically maps Header->Key, Row->Value
        let record: HashMap<String, String> = match result {
            Ok(rec) => rec,
            Err(e) => {
                eprintln!("Skipping bad row: {}", e);
                continue;
            }
        };

        // We removed the price filtering logic because we 
        // don't know if a "price" column exists in a generic file!
        clean_data.push(record);
    }

    let json_output = serde_json::to_string_pretty(&clean_data)?;
    fs::write(args.output, json_output)?;

    println!("Success! Converted {} records.", clean_data.len());
    Ok(())
}

Now, you can run this same tool on products.csv, or users.csv, or logs.csv. It has become a universal converter!

Step 5: The Grand Finale - Install It Globally

Right now, we are running our tool using cargo run. That’s fine for development, but in production, we want a standalone tool that we can run from anywhere—just like grep, jq, or python.

1. Build for Release

Cargo compiles in "debug" mode by default (which is fast to compile but slow to run). Let's build a highly optimized release binary.

cargo build --release

This creates a standalone executable file at ./target/release/rusty_pipe. You can literally email this file to a friend with the same OS, and it will run—they don't need Rust installed!

2. Install to your System Path

To make it available globally in your terminal, copy it to your bin folder.

For Mac/Linux:

sudo cp ./target/release/rusty_pipe /usr/local/bin/

For Windows: You can copy the .exe to any folder that is in your system PATH.

3. Run it like a Pro

Close your terminal, open a new one, navigate to your Desktop (or any folder with a CSV), and run:

rusty_pipe input.csv output.json

Congratulations! You just built and installed your own system-level CLI tool.

Conclusion

We built two versions of an ETL tool:

1. The Strict Version: Uses structs. Best for known data, ensures type safety, and allows easy filtering (e.g., price > 1.0).

2. The Generic Version: Uses HashMaps. Best for general-purpose utilities where the column names aren't known ahead of time.

This project covers the core concepts of Rust for Data Engineering: Cargo, Structs, Iterators, and Serde.