Case Study — Enterprise Data Platform

Enterprise Data Access.
Built to
scale enterprise intelligence.

DDLG is a secure, high-performance distributed data lake gateway. Connect disparate sources, execute visual workflows, and query billions of records with zero-cloud infrastructure.

100%
Org adoption
7+
Data transformers
Rust
AI engine runtime
1-click
Workflow execution
Scroll

See it in action.

A complete walkthrough of the DDLG platform — from connecting live data sources to executing a visual pipeline and querying the semantic Rust AI Engine.

Platform Demo — 4:12
CHAPTER 01
The Problem

The org was bleeding hours
to bad data operations.

These aren't hypotheticals. These were real conversations, real slowdowns, and real frustration that I witnessed and experienced firsthand. Here's what the day-to-day looked like before DDLG existed.

Data lived in silos

Finance had Oracle. Marketing had CSVs on a shared drive. Operations had MongoDB. No one could talk to each other — let alone search across all of them at once.

Processing was a full-time job

One analyst spent 3 hours every morning running a Python script to download an email attachment, clean it, and push it to a database. Every. Single. Day.

No reusable workflows

Six people had six versions of the same script. Each slightly different, each broken in different ways. When one changed, no one knew.

Dependencies created bottlenecks

Team A couldn't do their work until Team B ran their script. That dependency chain cascaded — one failure delayed the whole org for hours.

Search was impossible

"Where's that Q3 reconciliation file?" was a question asked at least five times a day across the org. No one had an answer. Everyone guessed.

AI was a distant dream

Cloud AI meant data leaving the firewall. The security team said no. So employees got nothing — while AI tools transformed every other industry.

I spent two weeks documenting every pain point. Then I sat down and asked:

"What would a single platform that solves all of this look like?"

CHAPTER 02
Architecture Decisions

Every decision was deliberate.
Here's the thinking.

I spent a week evaluating alternatives before writing a single line of code. Here are the five biggest architectural choices — what I considered, what I rejected, and exactly why I chose what I did.

01What runtime for the app itself?
Electron desktop app
02How to run AI inference locally without Python?
Rust-compiled AI daemon
03Where to store & query semantic vectors?
RocksDB embedded store
04How should users build data pipelines?
Guided visual pipeline builder
05How to handle multiple heterogeneous data sources?
Modular connector architecture

The guiding principle behind every call

Every decision was filtered through a single question: "Can a non-technical employee use this on day one without asking for help?" If no — back to the drawing board.

CHAPTER 03
What I Built

Six pillars.
Each solving a real pain.

Here's every major feature — the problem it solved, the approach I took, and the code that makes it work. Click each to dive deep.

FEATURE 01Independence

Multi-Source Data Connectors

Every employee connects to their own data. No waiting on anyone.

The biggest bottleneck wasn't processing — it was access. Team A waited on Team B to run a script to pull data from Oracle. Team C waited on someone else to download Gmail attachments. I broke all those dependencies by giving each user their own connector setup. Configure once, sync forever. Gmail, Oracle DB, MongoDB, and local file system — each a self-contained module.

Implementation highlights
→Gmail OAuth connector — scans inbox, downloads & indexes attachments automatically
→Oracle DB connector — query & stream structured rows into the local data layer
→MongoDB connector — flexible document ingestion for unstructured org data
→Local file system watcher — monitors a folder, re-indexes on any change
→Each connector runs independently — one failure never blocks others
implementation excerpt
// Connector interface — every source implements this
interface DataConnector {
  id: string;
  connect(config: ConnectorConfig): Promise<void>;
  sync(): AsyncGenerator<DataChunk>;
  onFileChanged(path: string): void;
}

// Gmail connector — runs in background thread
class GmailConnector implements DataConnector {
  async *sync() {
    const messages = await this.gmail.getAttachments();
    for (const msg of messages) {
      yield { source: 'gmail', content: msg, meta: msg.headers };
    }
  }
}
CHAPTER 04
System Architecture

A multi-process design
built for real workloads.

The Rust daemon, the indexing queue split, the IPC bridge — each layer has a clear responsibility. Here's how they stack and why they're separated the way they are.

Presentation Layer
React 18
Component UI + Hooks
Redux Toolkit
State management
Framer Motion
Animations
MUI + SCSS
Design system

The React renderer runs inside Electron's browser process. Redux manages global state (DT list, paths, pipeline state). All UI components are server-side agnostic — pure React.

Electron IPC Bridge
ipcRenderer
UI → Main calls
ipcMain
Main → UI events
preload.js
Secure context bridge
contextIsolation
Security boundary

The preload script exposes a typed API surface to the renderer. ipcRenderer.invoke() calls are type-checked. The main process never exposes Node.js APIs directly to the renderer context — it goes through the preload bridge.

Main Process (Node.js)
Data Connectors
Gmail · Oracle · MongoDB · FS
DT Manager
Transformer registry
Indexing Queues
Parse + AI queue separation
Daemon Manager
Rust process lifecycle

The Electron main process orchestrates everything. Separate indexing queues prevent AI generation from blocking file parsing. The DaemonManager spawns the Rust binary and maintains a stdio IPC channel to it.

Rust AI Daemon
Embedding Engine
Local model inference
File Parsers
PDF · CSV · XLSX · TXT
RocksDB Store
Vector persistence
IPC Protocol
stdio JSON-RPC

Compiled to a native binary — no interpreter. The daemon receives file paths over stdin, generates embeddings, and writes them to RocksDB. Vector queries return ranked results in <5ms. This entire AI layer runs offline, always.

The Indexing Queue Architecture

A key insight: parsing files and generating AI embeddings have very different performance profiles. I gave them separate queues — high-concurrency parsing doesn't block the rate-limited AI queue, and search remains live even during heavy sync.

File arrives (connector sync)
→
Parse Queue (high concurrency)
→
Text extracted + chunked
→
AI Queue (rate-limited)
→
Embeddings generated (Rust)
→
Stored in RocksDB
→
Searchable immediately
CHAPTER 05
The AI Engine

Local AI that's faster
than any cloud call.

The AI team said "use ChatGPT." I said no — data doesn't leave the building. So I built a Rust-compiled embedding daemon that runs the entire AI pipeline locally, at native machine speed, with zero cloud dependency.

No interpreter overhead

Rust compiles to a native binary. No JVM, no Python runtime, no V8 for the AI layer. Cold start is milliseconds.

Memory safety by default

No null pointer dereferences, no use-after-free. The embedding engine processes thousands of chunks without memory leaks — guaranteed by the compiler.

True parallelism with Rayon

Rayon's work-stealing thread pool saturates all CPU cores for parallel chunk embedding. A 100-page PDF embeds in under 200ms.

RocksDB native binding

The Rust-native RocksDB crate gives zero-copy access to the vector store. No serialization roundtrip — raw bytes in, raw bytes out.

How a file becomes searchable — step by step

Click any step to see more detail.

01Receive file path via stdin IPC
02Parse file (PDF/CSV/XLSX/TXT)
03Chunk text into overlapping windows
04Run local embedding model on each chunk
05Compute float32 vector (384 dimensions)
06Persist key → vector in RocksDB
07Emit done signal → Node main process
~3ms
Embedding latency
per document chunk, native Rust
<5ms
Vector query time
RocksDB cosine similarity
0 bytes
Cloud data sent
fully local, forever
∞
Files indexable
bound only by disk space
CHAPTER 06
Technology Choices

The full stack — and
why each piece is there.

No technology here was picked casually. Every choice has a justification rooted in the constraints: offline-first, employee-friendly, extensible.

Desktop Shell
Electron 28Cross-platform desktop app runtime

Native OS access, file system, IPC, custom window Chrome

Node.jsMain process runtime

Powers all background workers, connectors, and daemon management

Frontend
React 18UI component framework

Hooks, concurrent rendering, and a massive ecosystem

Redux ToolkitGlobal state management

DT list, paths, local files, pipeline state — all in a single predictable store

React Hook FormForm handling

Used across all DT input forms — zero re-renders, typed validation

Framer MotionAnimations

Every transition, panel swap, and micro-interaction

MUI + SCSSDesign system

MUI base components with a custom SCSS override layer for org branding

AI & Search
RustAI daemon runtime

Compiled to native binary — no interpreter, maximum throughput

RocksDBVector store

Embedded LSM-tree store — millions of keys, sub-millisecond reads, persistent

Local embedding modelSemantic vector generation

384-dimensional embeddings, runs entirely offline

Data Layer
Oracle DBEnterprise database connector

Used by Finance for reconciliation and reporting data

MongoDBDocument store connector

Product catalog, VR code lookups, unstructured data

Gmail API (OAuth)Email connector

Attachment ingestion, auto-sync, OAuth2 token management

Local File System (chokidar)FS watcher connector

Real-time directory watching, event-driven re-indexing

Build & Tooling
Webpack 5Bundle builder

Custom multi-entry config for Electron main + renderer process split

BabelJS/JSX transpilation

ES2020+ with React presets, custom Electron targets

Semantic ReleaseAutomated versioning

Commit-based changelogs and version bumps — no manual release management

electron-builderInstaller packaging

Creates .exe (Windows) and .AppImage (Linux) distributables from CI