ZeroIPC - Active Computational Substrate for Shared Memory

Overview

ZeroIPC transforms shared memory from passive storage into an active computational substrate, enabling both imperative and functional programming paradigms across process boundaries. It provides zero-copy data sharing with sophisticated concurrency primitives, reactive streams, and codata structures - bringing modern programming abstractions to inter-process communication.

Key Features

• 🚀 Zero-Copy Performance - Direct memory access without serialization
• 🌐 Language Independence - C++ and Python implementations, not bindings
• 🔒 Lock-Free Concurrency - Atomic operations and CAS-based algorithms
• 📦 Minimal Metadata - Only store name/offset/size for true flexibility
• 🦆 Duck Typing - Runtime type specification (Python) or compile-time templates (C++)
• 🎯 Simple Discovery - Named structures for easy cross-process lookup
• ⚡ Reactive Programming - Functional reactive streams with operators
• 🔮 Codata Support - Futures, lazy evaluation, and infinite streams
• 🚪 CSP Concurrency - Channels for synchronous message passing
• 🛠️ CLI Tools - Comprehensive inspection and debugging utilities

Quick Start

Basic Data Sharing

C++ Producer:

#include <zeroipc/memory.h>
#include <zeroipc/array.h>

// Create shared memory
zeroipc::Memory mem("/sensor_data", 10*1024*1024);  // 10MB

// Create typed array
zeroipc::Array<float> temps(mem, "temperature", 1000);
temps[0] = 23.5f;
Copy

Python Consumer:

from zeroipc import Memory, Array
import numpy as np

# Open same shared memory
mem = Memory("/sensor_data")

# Read with duck typing - user specifies type
temps = Array(mem, "temperature", dtype=np.float32)
print(temps[0])  # 23.5
Copy

Reactive Streams Example

Process A - Sensor Data Producer:

#include <zeroipc/memory.h>
#include <zeroipc/stream.h>

zeroipc::Memory mem("/sensors", 10*1024*1024);
zeroipc::Stream<double> temperature(mem, "temp_stream", 1000);

while (running) {
    double temp = read_sensor();
    temperature.emit(temp);
    std::this_thread::sleep_for(100ms);
}
Copy

Process B - Stream Processing:

zeroipc::Memory mem("/sensors");
zeroipc::Stream<double> temperature(mem, "temp_stream");

// Create derived streams with functional transformations
auto fahrenheit = temperature.map(mem, "temp_f", 
    [](double c) { return c * 9/5 + 32; });

auto warnings = fahrenheit.filter(mem, "warnings",
    [](double f) { return f > 100.0; });

// Subscribe to processed stream
warnings.subscribe([](double high_temp) {
    send_alert("High temperature: " + std::to_string(high_temp));
});
Copy

Futures for Async Results

Process A - Computation:

#include <zeroipc/future.h>

zeroipc::Memory mem("/compute", 10*1024*1024);
zeroipc::Future<double> result(mem, "expensive_calc");

// Perform expensive computation
double value = run_simulation();
result.set_value(value);
Copy

Process B - Waiting for Result:

zeroipc::Memory mem("/compute");
zeroipc::Future<double> result(mem, "expensive_calc", true);

// Wait with timeout
if (auto value = result.get_for(std::chrono::seconds(5))) {
    process_result(*value);
} else {
    handle_timeout();
}
Copy

CSP-Style Channels

Process A - Producer:

#include <zeroipc/channel.h>

zeroipc::Memory mem("/messages", 10*1024*1024);
zeroipc::Channel<Message> ch(mem, "commands", 100);  // buffered

Message msg{.type = CMD_START, .data = 42};
ch.send(msg);  // Blocks if buffer full
Copy

Process B - Consumer:

zeroipc::Memory mem("/messages");
zeroipc::Channel<Message> ch(mem, "commands");

while (auto msg = ch.receive()) {
    process_command(*msg);
}
Copy

Architecture

Binary Format Specification

All implementations follow the same binary format defined in SPECIFICATION.md:

[Table Header][Table Entries][Data Structure 1][Data Structure 2]...
Copy

• Table Header: Magic number, version, entry count, next offset
• Table Entry: Name (32 bytes), offset (4 bytes), size (4 bytes)
• Data Structures: Raw binary data, layout determined by structure type

Minimal Metadata Philosophy

Unlike traditional IPC systems, ZeroIPC stores NO type information:

• Name: For discovery
• Offset: Where data starts
• Size: How much memory is used

This enables true language independence:

• Both languages can create: Python and C++ can both allocate new structures
• Both languages can read: Either can discover and access existing structures
• Type safety per language: C++ uses templates, Python uses NumPy dtypes

Data Structures

Traditional Data Structures

• ✅ Array - Fixed-size contiguous storage with atomic operations
• ✅ Queue - Lock-free MPMC circular buffer using CAS
• ✅ Stack - Lock-free LIFO with ABA-safe operations
• ✅ Map - Lock-free hash map with linear probing
• ✅ Set - Lock-free hash set for unique elements
• ✅ Pool - Object pool with free list management
• ✅ Ring - High-performance ring buffer for streaming
• ✅ Table - Metadata registry for dynamic discovery

Codata & Computational Structures

• ✅ Future - Asynchronous computation results across processes
• ✅ Lazy - Deferred computations with automatic memoization
• ✅ Stream - Reactive data flows with FRP operators (map, filter, fold)
• ✅ Channel - CSP-style synchronous/buffered message passing

Why Codata?

Traditional data structures store values in space. Codata structures represent computations over time. This enables:

• Cross-process async/await - Future results shared between processes
• Lazy evaluation - Expensive computations cached and shared
• Reactive pipelines - Event-driven processing with backpressure
• CSP concurrency - Go-style channels for structured communication

See Codata Guide for detailed explanation.

Language Implementations

C Implementation

• Pure C99 for maximum portability
• Zero dependencies beyond POSIX
• Static library (libzeroipc.a)
• Minimal overhead

C++ Implementation

• Template-based for zero overhead
• Header-only library
• Modern C++23 features
• RAII resource management

Python Implementation

• Pure Python, no compilation required
• NumPy integration for performance
• Duck typing for flexibility
• mmap for direct memory access

Building and Testing

C

cd c
make            # Build library
make test       # Run tests
Copy

C++

cd cpp
cmake -B build .
cmake --build build
ctest --test-dir build
Copy

Python

cd python
pip install -e .
python -m pytest tests/
Copy

Cross-Language Tests

cd interop
./test_interop.sh          # C++ writes, Python reads
./test_reverse_interop.sh  # Python writes, C++ reads
Copy

Design Principles

1. Language Equality - No language is “primary”, all are first-class
2. Minimal Overhead - Table stores only what’s absolutely necessary
3. User Responsibility - Users ensure type consistency across languages
4. Zero Dependencies - Each implementation stands alone
5. Binary Compatibility - All languages read/write the same format

Performance

• Array Access: Identical to native arrays (zero overhead)
• Queue Operations: Lock-free with atomic CAS
• Memory Allocation: O(1) bump allocation
• Discovery: O(n) where n ≤ max_entries

Use Cases

ZeroIPC excels at:

• ✅ High-frequency sensor data sharing
• ✅ Multi-process simulations
• ✅ Real-time analytics pipelines
• ✅ Cross-language scientific computing
• ✅ Zero-copy producer-consumer patterns

Not designed for:

• ❌ General-purpose memory allocation
• ❌ Network-distributed systems
• ❌ Persistent storage
• ❌ Garbage collection

CLI Tools

zeroipc-inspect

Comprehensive tool for inspecting and debugging shared memory:

# Build the tool
cd cpp && cmake -B build . && cmake --build build
./build/tools/zeroipc-inspect

# List all ZeroIPC shared memory segments
./zeroipc-inspect list

# Show detailed information about a segment
./zeroipc-inspect show /sensor_data

# Monitor a stream in real-time
./zeroipc-inspect monitor /sensors temperature_stream

# Dump raw memory contents
./zeroipc-inspect dump /compute --offset 0 --size 1024
Copy

Documentation

• Codata Guide - Understanding codata and computational structures
• API Reference - Complete API documentation
• Architecture - System design and memory layout
• Design Patterns - Cross-process communication patterns
• CLI Tools - Command-line utilities documentation
• Examples - Complete working examples
• Design Philosophy - Core principles and trade-offs
• Binary Specification - Wire format all implementations follow
• C++ Documentation - C++ specific details
• Python Documentation - Python specific details

Contributing

Contributions welcome! When adding new language implementations:

1. Follow the binary specification exactly
2. Create a new directory for your language
3. Implement Memory, Table, and Array as minimum
4. Add cross-language tests in interop/

Advanced Features

Functional Programming in Shared Memory

ZeroIPC brings functional programming paradigms to IPC:

• Lazy Evaluation: Defer expensive computations until needed
• Memoization: Automatic caching of computation results
• Stream Combinators: map, filter, fold, take, skip, window
• Monadic Composition: Chain asynchronous operations with Futures

Cross-Process Patterns

• Producer-Consumer: Lock-free queues with backpressure
• Pub-Sub: Multiple consumers on reactive streams
• Request-Response: Futures for RPC-like patterns
• Pipeline: Stream transformations across processes
• Fork-Join: Parallel computation with result aggregation

Future Explorations

The boundary between data and code continues to blur:

• Persistent Data Structures: Immutable structures with structural sharing
• Software Transactional Memory: ACID transactions in shared memory
• Dataflow Programming: Computational graphs in shared memory
• Actors: Message-passing actors with mailboxes
• Continuations: Suspended computations for coroutines

License

MIT