gridkv

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 7, 2026 License: MIT Imports: 13 Imported by: 0

README

GridKV

Go Version GitHub License Go Report Card GitHub Tag

A high-performance distributed key-value cache with eventual consistency guarantees, using in-memory storage.

English / 中文

Features

  • High Performance & Thread-Safe: Sub-millisecond local operations, all public methods support concurrent access
  • Eventual Consistency Replication: Gossip protocol with batched replication ensures data convergence across nodes
  • Distributed Fault-Tolerant Architecture: Consistent hashing with virtual nodes and SWIM protocol for load balancing and failure detection
  • In-Memory Storage: Memory backend supports TTL and automatic compression

Installation

go get github.com/feellmoose/gridkv

Quick Start

package main

import (
    "context"
    "log"
    "time"
    "github.com/feellmoose/gridkv"
)

func main() {
    opts := &gridkv.GridKVOptions{
        LocalNodeID:  "node1",
        LocalAddress: "localhost:8080",
        SeedAddrs:    []string{"localhost:8081"},
    }

    kv, err := gridkv.NewGridKV(opts)
    if err != nil {
        log.Fatal(err)
    }
    defer kv.Close()

    // Set key-value pair
    kv.Set(context.Background(), "key", []byte("value"), time.Hour)

    // Get value
    // Returns (nil, nil) if key not found, error only for real failures
    value, err := kv.Get(context.Background(), "key")
    if err != nil {
        log.Fatal(err)
    }
    if value != nil {
        println(string(value))
    }
}

Testing

# Run all tests
go test ./...

# Run short tests only
go test -short ./...

# Run benchmarks
go test -bench=. ./tests/

Architecture

  • SWIM Protocol: Membership management and failure detection
  • Consistent Hashing: Key distribution across nodes
  • Gossip Protocol: Efficient data replication
  • HLC (Hybrid Logical Clock): Causality tracking and conflict resolution

See internal/cluster/README.md for detailed architecture documentation.

License

MIT License - see LICENSE for details.

Documentation

Overview

Package gridkv provides a distributed key-value cache with eventual consistency.

Features: consistent hashing, batched replication, SWIM failure detection, adaptive LAN/WAN networking, Prometheus/OTLP metrics.

Thread-safe: all public methods are safe for concurrent access.

Index

Constants

View Source 
const (
	LogLevelDebug = logging.LevelDebug
	LogLevelInfo  = logging.LevelInfo
	LogLevelWarn  = logging.LevelWarn
	LogLevelError = logging.LevelError
	LogLevelFatal = logging.LevelFatal
)

Logging level constants

View Source 
const (
	LogFormatText    = logging.FormatText
	LogFormatJSON    = logging.FormatJSON
	LogFormatCompact = logging.FormatCompact
)

Logging format constants

View Source 
const Version = "v0.3.8"

Version represents the current version of GridKV

Variables

View Source 
var (
	// ErrShuttingDown indicates the node has begun graceful shutdown
	ErrShuttingDown = errors.New("gridkv shutting down")

	// ErrVersionMismatch indicates a version conflict
	ErrVersionMismatch = errors.New("version mismatch")

	// ErrMemoryLimitExceeded indicates memory limit reached
	ErrMemoryLimitExceeded = errors.New("memory limit exceeded")

	// ErrEmptyKey indicates an empty key was provided
	ErrEmptyKey = mem_storage.ErrEmptyKey
)

Error constants exported for application error handling

Functions

This section is empty.

Types

type ClusterStats added in v0.3.4 

type ClusterStats struct {
	Ready         bool
	ClusterSize   int
	HealthyNodes  int
	ReplicaFactor int
	LocalNodeID   string
	PubkeysReady  bool
	PubkeyCount   int
	PeerCount     int
}

ClusterStats represents cluster health and readiness statistics.

type Config added in v0.4.0 

type Config struct {
	LocalNodeID  string   `json:"local_node_id"`
	LocalAddress string   `json:"local_address"`
	SeedAddrs    []string `json:"seed_addrs,omitempty"`

	TTL             string `json:"ttl,omitempty"` // Duration string (e.g., "5m")
	HotReadCacheTTL string `json:"hot_read_cache_ttl,omitempty"`
	HotCacheSize    int    `json:"hot_cache_size,omitempty"`
	VirtualNodes    int    `json:"virtual_nodes,omitempty"`
	ReplicaCount    int    `json:"replica_count,omitempty"`

	ReadRepairRateLimitPerSec int64 `json:"read_repair_rate_limit_per_sec,omitempty"`

	FailureTimeout     string `json:"failure_timeout,omitempty"`
	SuspectTimeout     string `json:"suspect_timeout,omitempty"`
	GossipInterval     string `json:"gossip_interval,omitempty"`
	ReplicationTimeout string `json:"replication_timeout,omitempty"`
	ReadTimeout        string `json:"read_timeout,omitempty"`
	StartupGracePeriod string `json:"startup_grace_period,omitempty"`
	DisableAuth        bool   `json:"disable_auth,omitempty"`

	BatchThreshold int    `json:"batch_threshold,omitempty"`
	BatchWindow    string `json:"batch_window,omitempty"`

	Network *NetworkConfig `json:"network,omitempty"`
	Storage *StorageConfig `json:"storage,omitempty"`
	Log     *LogConfig     `json:"log,omitempty"`
}

Config is used for JSON/YAML configuration file parsing with snake_case field names.

type GridKV 

type GridKV struct {
	// contains filtered or unexported fields
}

GridKV is the distributed key-value cache instance.

Components: cluster (membership/replication), storage backend, network transport. Thread-safe.

func NewGridKV 

func NewGridKV(options ...interface{}) (*GridKV, error)

NewGridKV initializes a GridKV instance with the provided options.

Example: 

kv, err := gridkv.NewGridKV(
	gridkv.WithLocalNodeID("node-1"),
	gridkv.WithLocalAddress("localhost:8080"),
	gridkv.WithSeedAddrs("localhost:8081"),
)

Required fields: LocalNodeID, LocalAddress

func (*GridKV) Close 

func (g *GridKV) Close(timeout ...time.Duration) error

Close shuts down GridKV: stops cluster (which manages all components via lifecycle). Uses 30s default timeout if none provided. Idempotent. Thread-safe. All resources (network, storage, executor, cache) are managed by cluster's lifecycle manager.

func (*GridKV) Delete 

func (g *GridKV) Delete(ctx context.Context, key string) (err error)

Delete removes a key-value pair.

Writes tombstone locally, enqueues async replication. Idempotent. Panic-safe, thread-safe.

func (*GridKV) Get 

func (g *GridKV) Get(ctx context.Context, key string) (value []byte, err error)

Get retrieves a value by key.

Reads locally if available, otherwise forwards to coordinator with retries. Returns freshest value, triggers read-repair on version mismatch. Returns deep copy.

If the key is not found, returns (nil, nil). Only returns an error for real failures (network errors, timeouts, etc.).

Panic-safe, thread-safe.

func (*GridKV) HealthCheck added in v0.1.2 

func (g *GridKV) HealthCheck() error

HealthCheck verifies GridKV is initialized and cluster has healthy nodes.

func (*GridKV) Set 

func (g *GridKV) Set(ctx context.Context, key string, value []byte, ttl ...time.Duration) (err error)

Set stores a key-value pair with eventual replication.

Computes replica set via consistent hashing, writes locally, enqueues async replication. Returns immediately. TTL overrides default (0 = no expiration). Panic-safe, thread-safe.

func (*GridKV) Stats added in v0.3.4 

func (g *GridKV) Stats() Stats

Stats returns complete GridKV statistics including cluster, network and storage stats. Thread-safe.

func (*GridKV) WaitReady added in v0.1.2 

func (g *GridKV) WaitReady(timeout time.Duration) error

WaitReady blocks until cluster is ready or timeout.

type GridKVOptions 

type GridKVOptions = Options

GridKVOptions is deprecated, use Options or functional options instead. Kept for backward compatibility.

type LogConfig added in v0.4.0 

type LogConfig struct {
	Level      string `json:"level,omitempty"`
	Format     string `json:"format,omitempty"`
	TimeFormat string `json:"time_format,omitempty"`
	NoCaller   bool   `json:"no_caller,omitempty"`
	NoTime     bool   `json:"no_time,omitempty"`
}

LogConfig is used for JSON/YAML configuration file parsing.

type LoggerOptions added in v0.3.0 

type LoggerOptions struct {
	Level      string
	Format     string
	Output     io.Writer
	TimeFormat string
	NoCaller   bool
	NoTime     bool
}

LoggerOptions configures logging (internal use).

type NetworkConfig added in v0.4.0 

type NetworkConfig struct {
	Type         string `json:"type,omitempty"` // "tcp" or "quic"
	BindAddr     string `json:"bind_addr,omitempty"`
	MaxIdle      int    `json:"max_idle,omitempty"`
	MaxConns     int    `json:"max_conns,omitempty"`
	Timeout      string `json:"timeout,omitempty"`       // Duration string
	ReadTimeout  string `json:"read_timeout,omitempty"`  // Duration string
	WriteTimeout string `json:"write_timeout,omitempty"` // Duration string
}

NetworkConfig is used for JSON/YAML configuration file parsing.

type NetworkOptions 

type NetworkOptions struct {
	Type         NetworkType
	BindAddr     string
	MaxIdle      int
	MaxConns     int
	Timeout      time.Duration
	ReadTimeout  time.Duration
	WriteTimeout time.Duration
}

NetworkOptions configures networking (internal use).

type NetworkType 

type NetworkType int

NetworkType represents the transport protocol. 

const (
	TCP  NetworkType = 1 // Reliable transport
	QUIC NetworkType = 3 // QUIC
)

type Option added in v0.4.0 

type Option func(*Options)

Option is a functional option for configuring GridKV.

func WithBatchThreshold added in v0.4.0 

func WithBatchThreshold(threshold int) Option

WithBatchThreshold sets the batch threshold.

func WithBatchWindow added in v0.4.0 

func WithBatchWindow(window time.Duration) Option

WithBatchWindow sets the batch window.

func WithDisableAuth added in v0.4.0 

func WithDisableAuth(disable bool) Option

WithDisableAuth disables authentication.

func WithFailureTimeout added in v0.4.0 

func WithFailureTimeout(timeout time.Duration) Option

WithFailureTimeout sets the failure timeout.

func WithGossipInterval added in v0.4.0 

func WithGossipInterval(interval time.Duration) Option

WithGossipInterval sets the gossip interval.

func WithHotCacheSize added in v0.4.0 

func WithHotCacheSize(size int) Option

WithHotCacheSize sets the hot read cache size.

func WithHotReadCacheTTL added in v0.4.0 

func WithHotReadCacheTTL(ttl time.Duration) Option

WithHotReadCacheTTL sets the hot read cache TTL.

func WithLocalAddress added in v0.4.0 

func WithLocalAddress(addr string) Option

WithLocalAddress sets the local address.

func WithLocalNodeID added in v0.4.0 

func WithLocalNodeID(nodeID string) Option

WithLocalNodeID sets the local node ID.

func WithLogger added in v0.4.0 

func WithLogger(log interface{}) Option

WithLogger sets the logger.

func WithLoggerOptions added in v0.4.0 

func WithLoggerOptions(opts LoggerOptions) Option

WithLoggerOptions sets the logger options.

func WithNetworkBindAddr added in v0.4.0 

func WithNetworkBindAddr(addr string) Option

WithNetworkBindAddr sets the network bind address.

func WithNetworkMaxConns added in v0.4.0 

func WithNetworkMaxConns(maxConns int) Option

WithNetworkMaxConns sets the maximum total connections per peer.

func WithNetworkMaxIdle added in v0.4.0 

func WithNetworkMaxIdle(maxIdle int) Option

WithNetworkMaxIdle sets the maximum idle connections per peer.

func WithNetworkReadTimeout added in v0.4.0 

func WithNetworkReadTimeout(timeout time.Duration) Option

WithNetworkReadTimeout sets the network read timeout.

func WithNetworkTimeout added in v0.4.0 

func WithNetworkTimeout(timeout time.Duration) Option

WithNetworkTimeout sets the network dial timeout.

func WithNetworkType added in v0.4.0 

func WithNetworkType(netType NetworkType) Option

WithNetworkType sets the network type.

func WithNetworkWriteTimeout added in v0.4.0 

func WithNetworkWriteTimeout(timeout time.Duration) Option

WithNetworkWriteTimeout sets the network write timeout.

func WithReadRepairRateLimit added in v0.4.0 

func WithReadRepairRateLimit(limit int64) Option

WithReadRepairRateLimit sets the read repair rate limit per second.

func WithReadTimeout added in v0.4.0 

func WithReadTimeout(timeout time.Duration) Option

WithReadTimeout sets the read timeout.

func WithReplicaCount added in v0.4.0 

func WithReplicaCount(count int) Option

WithReplicaCount sets the replica count.

func WithReplicationTimeout added in v0.4.0 

func WithReplicationTimeout(timeout time.Duration) Option

WithReplicationTimeout sets the replication timeout.

func WithSeedAddrs added in v0.4.0 

func WithSeedAddrs(addrs ...string) Option

WithSeedAddrs sets the seed addresses.

func WithStartupGracePeriod added in v0.4.0 

func WithStartupGracePeriod(period time.Duration) Option

WithStartupGracePeriod sets the startup grace period.

func WithStorageMaxMemoryMB added in v0.4.0 

func WithStorageMaxMemoryMB(mb int64) Option

WithStorageMaxMemoryMB sets the maximum memory in MB.

func WithStorageShardCount added in v0.4.0 

func WithStorageShardCount(count int) Option

WithStorageShardCount sets the shard count.

func WithSuspectTimeout added in v0.4.0 

func WithSuspectTimeout(timeout time.Duration) Option

WithSuspectTimeout sets the suspect timeout.

func WithTTL added in v0.4.0 

func WithTTL(ttl time.Duration) Option

WithTTL sets the default TTL.

func WithVirtualNodes added in v0.4.0 

func WithVirtualNodes(count int) Option

WithVirtualNodes sets the number of virtual nodes.

type Options added in v0.4.0 

type Options struct {
	LocalNodeID  string
	LocalAddress string
	SeedAddrs    []string

	TTL             time.Duration
	HotReadCacheTTL time.Duration
	HotCacheSize    int
	VirtualNodes    int
	ReplicaCount    int

	ReadRepairRateLimitPerSec int64

	FailureTimeout     time.Duration
	SuspectTimeout     time.Duration
	GossipInterval     time.Duration
	ReplicationTimeout time.Duration
	ReadTimeout        time.Duration
	StartupGracePeriod time.Duration
	DisableAuth        bool

	BatchThreshold int
	BatchWindow    time.Duration

	Network *NetworkOptions
	Storage *StorageOptions
	Log     interface{} // LoggerOptions, logging.Opts, or *logging.Logger
}

Options is the internal configuration structure used for GridKV initialization.

func ParseConfig added in v0.4.0 

func ParseConfig(data []byte) (*Options, error)

ParseConfig parses a JSON configuration with snake_case field names into Options.

type Stats added in v0.3.4 

type Stats struct {
	// Cluster stats
	Cluster ClusterStats

	// Network stats
	Network network.NetworkSnapshot

	// Storage stats
	Storage mem_storage.Stats

	// Version information
	Version string
}

Stats represents complete GridKV statistics including cluster, network and storage stats.

type StorageConfig added in v0.4.0 

type StorageConfig struct {
	MaxMemoryMB int64 `json:"max_memory_mb,omitempty"`
	ShardCount  int   `json:"shard_count,omitempty"`
}

StorageConfig is used for JSON/YAML configuration file parsing.

type StorageOptions 

type StorageOptions struct {
	MaxMemoryMB int64
	ShardCount  int
}

StorageOptions configures storage (internal use).

Source Files

View all Source files

Directories

Path Synopsis
internal
cluster
mem_storage
network
utils/bufferpool
utils/cache
utils/compress
utils/executor
utils/hlc
Package hlc implements Hybrid Logical Clock (HLC) for distributed timestamps.
 Package hlc implements Hybrid Logical Clock (HLC) for distributed timestamps.
utils/lifecycle
utils/logging
utils/zerocopy
tests
simulator
Package simulator provides test configurations for GridKV testing.
 Package simulator provides test configurations for GridKV testing.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.