nutsdb

package module
v1.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 1, 2025 License: Apache-2.0 Imports: 35 Imported by: 0

README

What is NutsDB?

English | 简体中文

NutsDB is a simple, fast, embeddable and persistent key/value store written in pure Go.

It supports fully serializable transactions and many data structures such as list、set、sorted set. All operations happen inside a Tx. Tx represents a transaction, which can be read-only or read-write. Read-only transactions can read values for a given bucket and a given key or iterate over a set of key-value pairs. Read-write transactions can read, update and delete keys from the DB.

We can learn more about NutsDB in details on the documents site of NutsDB: NutsDB Documents

Announcement

📢 Note: Starting from v0.9.0, defaultSegmentSize in DefaultOptions has been adjusted from 8MB to 256MB. The original value is the default value, which needs to be manually changed to 8MB, otherwise the original data will not be parsed. The reason for the size adjustment here is that there is a cache for file descriptors starting from v0.9.0 (detail see https://github.com/nutsdb/nutsdb/pull/164 ), so users need to look at the number of fds they use on the server, which can be set manually. If you have any questions, you can open an issue.

After nutsdb v1.0.0, due to changes in the underlying data storage protocol, the data of the old version is not compatible. Please rewrite it before using the new version. And the current Bucket needs to be created manually. Please see the Bucket usage documentation for details.

Architecture

nutsdb-架构图

Welcome contributions to NutsDB.

Quick start

Install NutsDB

To start using NutsDB, first needs Go installed (version 1.18+ is required). and run go get:

go get -u github.com/nutsdb/nutsdb

Opening a database

To open your database, use the nutsdb.Open() function,with the appropriate options.The Dir , EntryIdxMode and SegmentSize options are must be specified by the client. About options see here for detail.

package main

import (
    "log"

    "github.com/ZxwyProject/nutsdb"
)

func main() {
    // Open the database located in the /tmp/nutsdb directory.
    // It will be created if it doesn't exist.
    db, err := nutsdb.Open(
        nutsdb.DefaultOptions,
        nutsdb.WithDir("/tmp/nutsdb"),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    ...
}

Documentation

Buckets
Pairs
Iterator
Data Structures
Database Options
More Operation
Comparison
Benchmark

Contributors

Thank you for considering contributing to NutsDB! The contribution guide can be found in the CONTRIBUTING for details on submitting patches and the contribution workflow.

Acknowledgements

This package is inspired by the following:

License

The NutsDB is open-sourced software licensed under the Apache 2.0 license.

Documentation

Overview

Package nutsdb implements a simple, fast, embeddable and persistent key/value store written in pure Go. It supports fully serializable transactions. And it also supports data structure such as list、set、sorted set etc.

NutsDB currently works on Mac OS, Linux and Windows.

Usage

NutsDB has the following main types: DB, BPTree, Entry, DataFile And Tx. and NutsDB supports bucket, A bucket is a collection of unique keys that are associated with values.

All operations happen inside a Tx. Tx represents a transaction, which can be read-only or read-write. Read-only transactions can read values for a given key , or iterate over a set of key-value pairs (prefix scanning or range scanning). read-write transactions can also update and delete keys from the DB.

See the examples for more usage details.

Index

Constants

View Source 
const (
	IdSize = 8
	DsSize = 2
)
View Source 
const (
	MaxEntryHeaderSize = 4 + binary.MaxVarintLen32*3 + binary.MaxVarintLen64*3 + binary.MaxVarintLen16*3
	MinEntryHeaderSize = 4 + 9
)
View Source 
const (
	// UnCommitted represents the tx unCommitted status
	UnCommitted uint16 = 0

	// Committed represents the tx committed status
	Committed uint16 = 1
)
View Source 
const (
	B = 1

	KB = 1024 * B

	MB = 1024 * KB

	GB = 1024 * MB
)
View Source 
const (
	// BucketStatusExistAlready means this bucket already exists
	BucketStatusExistAlready = 1
	// BucketStatusDeleted means this bucket is already deleted
	BucketStatusDeleted = 2
	// BucketStatusNew means this bucket is created in current Tx
	BucketStatusNew = 3
	// BucketStatusUpdated means this bucket is updated in current Tx
	BucketStatusUpdated = 4
	// BucketStatusUnknown means this bucket doesn't exist
	BucketStatusUnknown = 5
)
View Source 
const (
	// SkipListMaxLevel represents the skipList max level number.
	SkipListMaxLevel = 32

	// SkipListP represents the p parameter of the skipList.
	SkipListP = 0.25
)
View Source 
const BucketStoreFileName = "bucket.Meta"
View Source 
const (
	// DataSuffix returns the data suffix
	DataSuffix = ".dat"
)
View Source 
const (
	DefaultMaxFileNums = 256
)
View Source 
const (
	DefaultThrottleSize = 16
)
View Source 
const FLockName = "nutsdb-flock"
View Source 
const KvWriteChCapacity = 1000
View Source 
const MAX_SIZE = math.MaxUint32
View Source 
const Persistent uint32 = 0

Persistent represents the data persistent flag

View Source 
const ScanNoLimit int = -1

ScanNoLimit represents the data scan no limit flag

View Source 
const SeparatorForListKey = "|"

SeparatorForListKey represents separator for listKey

View Source 
const SeparatorForZSetKey = "|"

SeparatorForZSetKey represents separator for zSet key.

View Source 
const (
	TooManyFileOpenErrSuffix = "too many open files"
)

Variables

View Source 
var (
	// ErrCrc is returned when crc is error
	ErrCrc = errors.New("crc error")

	// ErrCapacity is returned when capacity is error.
	ErrCapacity = errors.New("capacity error")

	ErrEntryZero = errors.New("entry is zero ")
)
View Source 
var (
	// ErrDBClosed is returned when db is closed.
	ErrDBClosed = errors.New("db is closed")

	// ErrBucket is returned when bucket is not in the HintIdx.
	ErrBucket = errors.New("err bucket")

	// ErrFn is returned when fn is nil.
	ErrFn = errors.New("err fn")

	// ErrBucketNotFound is returned when looking for bucket that does not exist
	ErrBucketNotFound = errors.New("bucket not found")

	// ErrDataStructureNotSupported is returned when pass a not supported data structure
	ErrDataStructureNotSupported = errors.New("this data structure is not supported for now")

	// ErrDirLocked is returned when can't get the file lock of dir
	ErrDirLocked = errors.New("the dir of db is locked")

	// ErrDirUnlocked is returned when the file lock already unlocked
	ErrDirUnlocked = errors.New("the dir of db is unlocked")

	// ErrIsMerging is returned when merge in progress
	ErrIsMerging = errors.New("merge in progress")

	// ErrNotSupportMergeWhenUsingList is returned calling 'Merge' when using list
	ErrNotSupportMergeWhenUsingList = errors.New("not support merge when using list for now")

	// ErrRecordIsNil is returned when Record is nil
	ErrRecordIsNil = errors.New("the record is nil")
)
View Source 
var (
	ErrPayLoadSizeMismatch   = errors.New("the payload size in Meta mismatch with the payload size needed")
	ErrHeaderSizeOutOfBounds = errors.New("the header size is out of bounds")
)
View Source 
var (
	// ErrListNotFound is returned when the list not found.
	ErrListNotFound = errors.New("the list not found")

	// ErrCount is returned when count is error.
	ErrCount = errors.New("err count")

	// ErrEmptyList is returned when the list is empty.
	ErrEmptyList = errors.New("the list is empty")

	// ErrStartOrEnd is returned when start > end
	ErrStartOrEnd = errors.New("start or end error")
)
View Source 
var (
	// ErrUnmappedMemory is returned when a function is called on unmapped memory
	ErrUnmappedMemory = errors.New("unmapped memory")

	// ErrIndexOutOfBound is returned when given offset out of mapped region
	ErrIndexOutOfBound = errors.New("offset out of mapped region")
)
View Source 
var (
	// ErrSetNotExist is returned when the key does not exist.
	ErrSetNotExist = errors.New("set not exist")

	// ErrSetMemberNotExist is returned when the member of set does not exist
	ErrSetMemberNotExist = errors.New("set member not exist")

	// ErrMemberEmpty is returned when the item received is nil
	ErrMemberEmpty = errors.New("item empty")
)
View Source 
var (
	ErrSortedSetNotFound = errors.New("the sortedSet does not exist")

	ErrSortedSetMemberNotExist = errors.New("the member of sortedSet does not exist")

	ErrSortedSetIsEmpty = errors.New("the sortedSet if empty")
)
View Source 
var (
	// ErrDataSizeExceed is returned when given key and value size is too big.
	ErrDataSizeExceed = errors.New("data size too big")

	// ErrTxClosed is returned when committing or rolling back a transaction
	// that has already been committed or rolled back.
	ErrTxClosed = errors.New("tx is closed")

	// ErrTxNotWritable is returned when performing a write operation on
	// a read-only transaction.
	ErrTxNotWritable = errors.New("tx not writable")

	// ErrKeyEmpty is returned if an empty key is passed on an update function.
	ErrKeyEmpty = errors.New("key cannot be empty")

	// ErrBucketEmpty is returned if bucket is empty.
	ErrBucketEmpty = errors.New("bucket is empty")

	// ErrRangeScan is returned when range scanning not found the result
	ErrRangeScan = errors.New("range scans not found")

	// ErrPrefixScan is returned when prefix scanning not found the result
	ErrPrefixScan = errors.New("prefix scans not found")

	// ErrPrefixSearchScan is returned when prefix and search scanning not found the result
	ErrPrefixSearchScan = errors.New("prefix and search scans not found")

	// ErrNotFoundKey is returned when key not found int the bucket on an view function.
	ErrNotFoundKey = errors.New("key not found in the bucket")

	// ErrCannotCommitAClosedTx is returned when the tx committing a closed tx
	ErrCannotCommitAClosedTx = errors.New("can not commit a closed tx")

	// ErrCannotRollbackACommittingTx is returned when the tx rollback a committing tx
	ErrCannotRollbackACommittingTx = errors.New("can not rollback a committing tx")

	ErrCannotRollbackAClosedTx = errors.New("can not rollback a closed tx")

	// ErrNotFoundBucket is returned when key not found int the bucket on an view function.
	ErrNotFoundBucket = errors.New("bucket not found")

	// ErrTxnTooBig is returned if too many writes are fit into a single transaction.
	ErrTxnTooBig = errors.New("Txn is too big to fit into one request")

	// ErrTxnExceedWriteLimit is returned when this tx's write is exceed max write record
	ErrTxnExceedWriteLimit = errors.New("txn is exceed max write record count")

	ErrBucketAlreadyExist = errors.New("bucket is already exist")

	ErrorBucketNotExist = errors.New("bucket is not exist yet, please use NewBucket function to create this bucket first")

	ErrValueNotInteger = errors.New("value is not an integer")

	ErrOffsetInvalid = errors.New("offset is invalid")

	ErrKVArgsLenNotEven = errors.New("parameters is used to represent key value pairs and cannot be odd numbers")

	ErrStartGreaterThanEnd = errors.New("start is greater than end")
)
View Source 
var BucketMetaSize int64
View Source 
var DefaultOptions = func() Options {
	return Options{
		EntryIdxMode:              HintKeyValAndRAMIdxMode,
		SegmentSize:               defaultSegmentSize,
		NodeNum:                   1,
		RWMode:                    FileIO,
		SyncEnable:                true,
		CommitBufferSize:          4 * MB,
		MergeInterval:             2 * time.Hour,
		MaxBatchSize:              (15 * defaultSegmentSize / 4) / 100,
		MaxBatchCount:             (15 * defaultSegmentSize / 4) / 100 / 100,
		HintKeyAndRAMIdxCacheSize: 0,
		ExpiredDeleteType:         TimeWheel,
	}
}()

DefaultOptions represents the default options.

View Source 
var ErrBucketCrcInvalid = errors.New("bucket crc invalid")
View Source 
var ErrBucketNotExist = errors.New("bucket not exist")
View Source 
var ErrCommitAfterFinish = errors.New("batch commit not permitted after finish")

ErrCommitAfterFinish indicates that write batch commit was called after

View Source 
var ErrDontNeedMerge = errors.New("the number of files waiting to be merged is at least 2")
View Source 
var ErrKeyNotFound = ErrNotFoundKey

ErrKeyNotFound is returned when the key is not in the b tree.

View Source 
var ErrSeparatorForListKey = errors.Errorf("contain separator (%s) for List key", SeparatorForListKey)

ErrSeparatorForListKey returns when list key contains the SeparatorForListKey.

Functions

func ConvertBigEndianBytesToUint64 

func ConvertBigEndianBytesToUint64(data []byte) uint64

func ConvertUint64ToBigEndianBytes 

func ConvertUint64ToBigEndianBytes(value uint64) []byte

func ErrBucketAndKey 

func ErrBucketAndKey(bucket string, key []byte) error

ErrBucketAndKey returns when bucket or key not found.

func ErrNotFoundKeyInBucket 

func ErrNotFoundKeyInBucket(bucket string, key []byte) error

ErrNotFoundKeyInBucket returns when key not in the bucket.

func ErrSeparatorForZSetKey 

func ErrSeparatorForZSetKey() error

ErrSeparatorForZSetKey returns when zSet key contains the SeparatorForZSetKey flag.

func GetDiskSizeFromSingleObject 

func GetDiskSizeFromSingleObject(obj interface{}) int64

func GetRandomBytes 

func GetRandomBytes(length int) []byte

func GetTestBytes 

func GetTestBytes(i int) []byte

func IsBucketEmpty 

func IsBucketEmpty(err error) bool

IsBucketEmpty is true if the bucket is empty.

func IsBucketNotFound 

func IsBucketNotFound(err error) bool

IsBucketNotFound is true if the error indicates the bucket is not exists.

func IsDBClosed 

func IsDBClosed(err error) bool

IsDBClosed is true if the error indicates the db was closed.

func IsExpired 

func IsExpired(ttl uint32, timestamp uint64) bool

IsExpired checks the ttl if expired or not.

func IsKeyEmpty 

func IsKeyEmpty(err error) bool

IsKeyEmpty is true if the key is empty.

func IsKeyNotFound 

func IsKeyNotFound(err error) bool

IsKeyNotFound is true if the error indicates the key is not found.

func IsPrefixScan 

func IsPrefixScan(err error) bool

IsPrefixScan is true if prefix scanning not found the result.

func IsPrefixSearchScan 

func IsPrefixSearchScan(err error) bool

IsPrefixSearchScan is true if prefix and search scanning not found the result.

func MarshalInts 

func MarshalInts(ints []int) ([]byte, error)

func MatchForRange 

func MatchForRange(pattern, bucket string, f func(bucket string) bool) (end bool, err error)

func OneOfUint16Array 

func OneOfUint16Array(value uint16, array []uint16) bool

func Truncate 

func Truncate(path string, capacity int64, f *os.File) error

Truncate changes the size of the file.

func UnmarshalInts 

func UnmarshalInts(data []byte) ([]int, error)

func UvarintSize 

func UvarintSize(x uint64) int

Types

type BTree 

type BTree struct {
	// contains filtered or unexported fields
}

func NewBTree 

func NewBTree() *BTree

func (*BTree) All 

func (bt *BTree) All() []*Record

func (*BTree) AllItems 

func (bt *BTree) AllItems() []*Item

func (*BTree) Count 

func (bt *BTree) Count() int

func (*BTree) Delete 

func (bt *BTree) Delete(key []byte) bool

func (*BTree) Find 

func (bt *BTree) Find(key []byte) (*Record, bool)

func (*BTree) Insert 

func (bt *BTree) Insert(record *Record) bool

func (*BTree) InsertRecord 

func (bt *BTree) InsertRecord(key []byte, record *Record) bool

func (*BTree) Max 

func (bt *BTree) Max() (*Item, bool)

func (*BTree) Min 

func (bt *BTree) Min() (*Item, bool)

func (*BTree) PopMax 

func (bt *BTree) PopMax() (*Item, bool)

func (*BTree) PopMin 

func (bt *BTree) PopMin() (*Item, bool)

func (*BTree) PrefixScan 

func (bt *BTree) PrefixScan(prefix []byte, offset, limitNum int) []*Record

func (*BTree) PrefixSearchScan 

func (bt *BTree) PrefixSearchScan(prefix []byte, reg string, offset, limitNum int) []*Record

func (*BTree) Range 

func (bt *BTree) Range(start, end []byte) []*Record

type BTreeIdx 

type BTreeIdx struct {
	// contains filtered or unexported fields
}

type Bucket 

type Bucket struct {
	// Meta: the metadata for this bucket
	Meta *BucketMeta
	// Id: is the marker for this bucket, every bucket creation activity will generate a new Id for it.
	// for example. If you have a bucket called "bucket_1", and you just delete bucket and create it again.
	// the last bucket will have a different Id from the previous one.
	Id BucketId
	// Ds: the data structure for this bucket. (List, Set, SortSet, String)
	Ds Ds
	// Name: the name of this bucket.
	Name string
}

Bucket is the disk structure of bucket

func (*Bucket) Decode 

func (b *Bucket) Decode(bytes []byte) error

Decode : Meta | BucketId | Ds | BucketName

func (*Bucket) Encode 

func (b *Bucket) Encode() []byte

Encode : Meta | BucketId | Ds | BucketName

func (*Bucket) GetCRC 

func (b *Bucket) GetCRC(headerBuf []byte, dataBuf []byte) uint32

func (*Bucket) GetEntrySize 

func (b *Bucket) GetEntrySize() int

func (*Bucket) GetPayloadSize 

func (b *Bucket) GetPayloadSize() int

type BucketId 

type BucketId = uint64

type BucketManager 

type BucketManager struct {

	// BucketInfoMapper BucketID => Bucket itself
	BucketInfoMapper InfoMapperInBucket

	BucketIDMarker IDMarkerInBucket

	// IDGenerator helps generates an ID for every single bucket
	Gen *IDGenerator
	// contains filtered or unexported fields
}

func NewBucketManager 

func NewBucketManager(dir string) (*BucketManager, error)

func (*BucketManager) ExistBucket 

func (bm *BucketManager) ExistBucket(ds Ds, name BucketName) bool

func (*BucketManager) GetBucket 

func (bm *BucketManager) GetBucket(ds Ds, name BucketName) (b *Bucket, err error)

func (*BucketManager) GetBucketById 

func (bm *BucketManager) GetBucketById(id BucketId) (*Bucket, error)

func (*BucketManager) GetBucketID 

func (bm *BucketManager) GetBucketID(ds Ds, name BucketName) (BucketId, error)

func (*BucketManager) SubmitPendingBucketChange 

func (bm *BucketManager) SubmitPendingBucketChange(reqs []*bucketSubmitRequest) error

type BucketMeta 

type BucketMeta struct {
	Crc uint32
	// Op: Mark the latest operation (e.g. delete, insert, update) for this bucket.
	Op BucketOperation
	// Size: the size of payload.
	Size uint32
}

BucketMeta stores the Meta info of a Bucket. E.g. the size of bucket it store in disk.

func (*BucketMeta) Decode 

func (meta *BucketMeta) Decode(bytes []byte)

Decode : CRC | op | size

type BucketName 

type BucketName = string

type BucketOperation 

type BucketOperation uint16
const (
	BucketInsertOperation BucketOperation = 1
	BucketUpdateOperation BucketOperation = 2
	BucketDeleteOperation BucketOperation = 3
)

type BucketStatus 

type BucketStatus = uint8

BucketStatus represents the current status of bucket in current Tx

type CEntries 

type CEntries struct {
	Entries
	LessFunc func(l, r string) bool
}

func (CEntries) Len 

func (c CEntries) Len() int

func (CEntries) Less 

func (c CEntries) Less(i, j int) bool

func (CEntries) Swap 

func (c CEntries) Swap(i, j int)

type DB 

type DB struct {
	Index      *index
	ActiveFile *DataFile
	MaxFileID  int64

	KeyCount int // total key number ,include expired, deleted, repeated.

	RecordCount int64 // current valid record count, exclude deleted, repeated
	// contains filtered or unexported fields
}

DB represents a collection of buckets that persist on disk.

func Open 

func Open(options Options, ops ...Option) (*DB, error)

Open returns a newly initialized DB object with Option.

func (*DB) Backup 

func (db *DB) Backup(dir string) error

Backup copies the database to file directory at the given dir.

func (*DB) BackupTarGZ 

func (db *DB) BackupTarGZ(w io.Writer) error

BackupTarGZ Backup copy the database to writer.

func (*DB) Begin 

func (db *DB) Begin(writable bool) (tx *Tx, err error)

Begin opens a new transaction. Multiple read-only transactions can be opened at the same time but there can only be one read/write transaction at a time. Attempting to open a read/write transactions while another one is in progress will result in blocking until the current read/write transaction is completed. All transactions must be closed by calling Commit() or Rollback() when done.

func (*DB) Close 

func (db *DB) Close() error

Close releases all db resources.

func (*DB) IsClose 

func (db *DB) IsClose() bool

IsClose return the value that represents the status of DB

func (*DB) Merge 

func (db *DB) Merge() error

func (*DB) NewWriteBatch 

func (db *DB) NewWriteBatch() (*WriteBatch, error)

func (*DB) Update 

func (db *DB) Update(fn func(tx *Tx) error) error

Update executes a function within a managed read/write transaction.

func (*DB) View 

func (db *DB) View(fn func(tx *Tx) error) error

View executes a function within a managed read-only transaction.

type DataFile 

type DataFile struct {
	ActualSize int64
	// contains filtered or unexported fields
}

DataFile records about data file information.

func NewDataFile 

func NewDataFile(path string, rwManager RWManager) *DataFile

NewDataFile will return a new DataFile Object.

func (*DataFile) Close 

func (df *DataFile) Close() (err error)

Close closes the RWManager. If RWManager is FileRWManager represents closes the File, rendering it unusable for I/O. If RWManager is a MMapRWManager represents Unmap deletes the memory mapped region, flushes any remaining changes.

func (*DataFile) ReadEntry 

func (df *DataFile) ReadEntry(off int, payloadSize int64) (e *Entry, err error)

ReadEntry returns entry at the given off(offset). payloadSize = bucketSize + keySize + valueSize

func (*DataFile) Release 

func (df *DataFile) Release() (err error)

func (*DataFile) Sync 

func (df *DataFile) Sync() (err error)

Sync commits the current contents of the file to stable storage. Typically, this means flushing the file system's in-memory copy of recently written data to disk.

func (*DataFile) WriteAt 

func (df *DataFile) WriteAt(b []byte, off int64) (n int, err error)

WriteAt copies data to mapped region from the b slice starting at given off and returns number of bytes copied to the mapped region.

type DataFlag 

type DataFlag = uint16

DataFlag means the data operations have done by users. 

const (
	// DataDeleteFlag represents the data delete flag
	DataDeleteFlag DataFlag = 0

	// DataSetFlag represents the data set flag
	DataSetFlag DataFlag = 1

	// DataLPushFlag represents the data LPush flag
	DataLPushFlag DataFlag = 2

	// DataRPushFlag represents the data RPush flag
	DataRPushFlag DataFlag = 3

	// DataLRemFlag represents the data LRem flag
	DataLRemFlag DataFlag = 4

	// DataLPopFlag represents the data LPop flag
	DataLPopFlag DataFlag = 5

	// DataRPopFlag represents the data RPop flag
	DataRPopFlag DataFlag = 6

	// DataLTrimFlag represents the data LTrim flag
	DataLTrimFlag DataFlag = 8

	// DataZAddFlag represents the data ZAdd flag
	DataZAddFlag DataFlag = 9

	// DataZRemFlag represents the data ZRem flag
	DataZRemFlag DataFlag = 10

	// DataZRemRangeByRankFlag represents the data ZRemRangeByRank flag
	DataZRemRangeByRankFlag DataFlag = 11

	// DataZPopMaxFlag represents the data ZPopMax flag
	DataZPopMaxFlag DataFlag = 12

	// DataZPopMinFlag represents the data aZPopMin flag
	DataZPopMinFlag DataFlag = 13

	// DataSetBucketDeleteFlag represents the delete Set bucket flag
	DataSetBucketDeleteFlag DataFlag = 14

	// DataSortedSetBucketDeleteFlag represents the delete Sorted Set bucket flag
	DataSortedSetBucketDeleteFlag DataFlag = 15

	// DataBTreeBucketDeleteFlag represents the delete BTree bucket flag
	DataBTreeBucketDeleteFlag DataFlag = 16

	// DataListBucketDeleteFlag represents the delete List bucket flag
	DataListBucketDeleteFlag DataFlag = 17

	// DataLRemByIndex represents the data LRemByIndex flag
	DataLRemByIndex DataFlag = 18

	// DataExpireListFlag represents that set ttl for the list
	DataExpireListFlag DataFlag = 19
)

type DataStatus 

type DataStatus = uint16

DataStatus means the status of data

type DataStructure 

type DataStructure = uint16

DataStructure represents the data structure we have already supported 

const (
	// DataStructureSet represents the data structure set flag
	DataStructureSet DataStructure = 0

	// DataStructureSortedSet represents the data structure sorted set flag
	DataStructureSortedSet DataStructure = 1

	// DataStructureBTree represents the data structure b tree flag
	DataStructureBTree DataStructure = 2

	// DataStructureList represents the data structure list flag
	DataStructureList DataStructure = 3
)

type Ds 

type Ds = uint16

type Entries 

type Entries []*Entry

Entries represents entries

func (Entries) Len 

func (e Entries) Len() int

func (Entries) Less 

func (e Entries) Less(i, j int) bool

func (Entries) Swap 

func (e Entries) Swap(i, j int)

func (Entries) ToCEntries 

func (e Entries) ToCEntries(lFunc func(l, r string) bool) CEntries

type Entry 

type Entry struct {
	Key   []byte
	Value []byte
	Meta  *MetaData
}

Entry represents the data item.

func NewEntry 

func NewEntry() *Entry

NewEntry new Entry Object

func (*Entry) Encode 

func (e *Entry) Encode() []byte

Encode returns the slice after the entry be encoded. 

the entry stored format:
|----------------------------------------------------------------------------------------------------------|
|  crc  | timestamp | ksz | valueSize | flag  | TTL  | status | ds   | txId |  bucketId |  key  | value    |
|----------------------------------------------------------------------------------------------------------|
| uint32| uint64  |uint32 |  uint32 | uint16  | uint32| uint16 | uint16 |uint64 | uint64 | []byte | []byte |
|----------------------------------------------------------------------------------------------------------|

func (*Entry) GetCrc 

func (e *Entry) GetCrc(buf []byte) uint32

GetCrc returns the crc at given buf slice.

func (*Entry) GetTxIDBytes 

func (e *Entry) GetTxIDBytes() []byte

GetTxIDBytes return the bytes of TxID

func (*Entry) IsBelongsToBPlusTree 

func (e *Entry) IsBelongsToBPlusTree() bool

func (*Entry) IsBelongsToList 

func (e *Entry) IsBelongsToList() bool

func (*Entry) IsBelongsToSet 

func (e *Entry) IsBelongsToSet() bool

func (*Entry) IsBelongsToSortSet 

func (e *Entry) IsBelongsToSortSet() bool

func (*Entry) IsZero 

func (e *Entry) IsZero() bool

IsZero checks if the entry is zero or not.

func (*Entry) ParseMeta 

func (e *Entry) ParseMeta(buf []byte) (int64, error)

ParseMeta parse Meta object to entry

func (*Entry) ParsePayload 

func (e *Entry) ParsePayload(data []byte) error

ParsePayload means this function will parse a byte array to bucket, key, size of an entry

func (*Entry) Size 

func (e *Entry) Size() int64

Size returns the size of the entry.

func (*Entry) WithKey 

func (e *Entry) WithKey(key []byte) *Entry

WithKey set key to Entry

func (*Entry) WithMeta 

func (e *Entry) WithMeta(meta *MetaData) *Entry

WithMeta set Meta to Entry

func (*Entry) WithValue 

func (e *Entry) WithValue(value []byte) *Entry

WithValue set value to Entry

type EntryIdxMode 

type EntryIdxMode int

EntryIdxMode represents entry index mode. 

const (
	// HintKeyValAndRAMIdxMode represents ram index (key and value) mode.
	HintKeyValAndRAMIdxMode EntryIdxMode = iota

	// HintKeyAndRAMIdxMode represents ram index (only key) mode.
	HintKeyAndRAMIdxMode
)

type EntryStatus 

type EntryStatus = uint8

EntryStatus represents the Entry status in the current Tx 

const (
	// NotFoundEntry means there is no changes for this entry in current Tx
	NotFoundEntry EntryStatus = 0
	// EntryDeleted means this Entry has been deleted in the current Tx
	EntryDeleted EntryStatus = 1
	// EntryUpdated means this Entry has been updated in the current Tx
	EntryUpdated EntryStatus = 2
)

type EntryWhenRecovery 

type EntryWhenRecovery struct {
	Entry
	// contains filtered or unexported fields
}

type ErrorHandler 

type ErrorHandler interface {
	HandleError(err error)
}

An ErrorHandler handles an error occurred during transaction.

type ErrorHandlerFunc 

type ErrorHandlerFunc func(err error)

The ErrorHandlerFunc type is an adapter to ErrorHandler.

func (ErrorHandlerFunc) HandleError 

func (fn ErrorHandlerFunc) HandleError(err error)

type ExpiredDeleteType 

type ExpiredDeleteType uint8
const (
	// TimeWheel represents use time wheel to do expired deletion
	TimeWheel ExpiredDeleteType = iota

	// TimeHeap represents use time heap to do expired deletion
	TimeHeap
)

type FdInfo 

type FdInfo struct {
	// contains filtered or unexported fields
}

FdInfo holds base fd info

type FileIORWManager 

type FileIORWManager struct {
	// contains filtered or unexported fields
}

FileIORWManager represents the RWManager which using standard I/O.

func (*FileIORWManager) Close 

func (fm *FileIORWManager) Close() (err error)

Close will remove the cache in the fdm of the specified path, and call the close method of the os of the file

func (*FileIORWManager) ReadAt 

func (fm *FileIORWManager) ReadAt(b []byte, off int64) (n int, err error)

ReadAt reads len(b) bytes from the File starting at byte offset off. `ReadAt` is a wrapper of the *File.ReadAt.

func (*FileIORWManager) Release 

func (fm *FileIORWManager) Release() (err error)

Release is a wrapper around the reduceUsing method

func (*FileIORWManager) Size 

func (fm *FileIORWManager) Size() int64

func (*FileIORWManager) Sync 

func (fm *FileIORWManager) Sync() (err error)

Sync commits the current contents of the file to stable storage. Typically, this means flushing the file system's in-memory copy of recently written data to disk. `Sync` is a wrapper of the *File.Sync.

func (*FileIORWManager) WriteAt 

func (fm *FileIORWManager) WriteAt(b []byte, off int64) (n int, err error)

WriteAt writes len(b) bytes to the File starting at byte offset off. `WriteAt` is a wrapper of the *File.WriteAt.

type GetByScoreRangeOptions 

type GetByScoreRangeOptions struct {
	Limit        int  // limit the max nodes to return
	ExcludeStart bool // exclude start value, so it search in interval (start, end] or (start, end)
	ExcludeEnd   bool // exclude end value, so it search in interval [start, end) or (start, end)
}

GetByScoreRangeOptions represents the options of the GetByScoreRange function.

type HeadTailSeq 

type HeadTailSeq struct {
	Head uint64
	Tail uint64
}

HeadTailSeq list head and tail seq num

type IDGenerator 

type IDGenerator struct {
	// contains filtered or unexported fields
}

func (*IDGenerator) CompareAndSetMaxId 

func (g *IDGenerator) CompareAndSetMaxId(id uint64)

func (*IDGenerator) GenId 

func (g *IDGenerator) GenId() uint64

type IDMarkerInBucket 

type IDMarkerInBucket map[BucketName]map[Ds]BucketId

type IdxType 

type IdxType interface {
	BTree | Set | SortedSet | List
}

type InfoMapperInBucket 

type InfoMapperInBucket map[BucketId]*Bucket

type Item 

type Item struct {
	// contains filtered or unexported fields
}

type Iterator 

type Iterator struct {
	// contains filtered or unexported fields
}

func NewIterator 

func NewIterator(tx *Tx, bucket string, options IteratorOptions) *Iterator

func (*Iterator) Key 

func (it *Iterator) Key() []byte

func (*Iterator) Next 

func (it *Iterator) Next() bool

func (*Iterator) Rewind 

func (it *Iterator) Rewind() bool

func (*Iterator) Seek 

func (it *Iterator) Seek(key []byte) bool

func (*Iterator) Valid 

func (it *Iterator) Valid() bool

func (*Iterator) Value 

func (it *Iterator) Value() ([]byte, error)

type IteratorOptions 

type IteratorOptions struct {
	Reverse bool
}

type LRUCache 

type LRUCache struct {
	// contains filtered or unexported fields
}

LRUCache is a least recently used (LRU) cache.

func NewLruCache 

func NewLruCache(cap int) *LRUCache

New creates a new LRUCache with the specified capacity.

func (*LRUCache) Add 

func (c *LRUCache) Add(key interface{}, value interface{})

Add adds a new entry to the cache.

func (*LRUCache) Clear 

func (c *LRUCache) Clear()

Clear clears the cache.

func (*LRUCache) Get 

func (c *LRUCache) Get(key interface{}) interface{}

Get returns the entry associated with the given key, or nil if the key is not in the cache.

func (*LRUCache) Len 

func (c *LRUCache) Len() int

Len returns the number of entries in the cache.

func (*LRUCache) Remove 

func (c *LRUCache) Remove(key interface{})

Remove removes the entry associated with the given key from the cache.

type LessFunc 

type LessFunc func(l, r string) bool

type List 

type List struct {
	Items     map[string]*BTree
	TTL       map[string]uint32
	TimeStamp map[string]uint64
	Seq       map[string]*HeadTailSeq
}

List represents the list.

func NewList 

func NewList() *List

func (*List) GetListTTL 

func (l *List) GetListTTL(key string) (uint32, error)

func (*List) IsEmpty 

func (l *List) IsEmpty(key string) (bool, error)

func (*List) IsExpire 

func (l *List) IsExpire(key string) bool

func (*List) LPeek 

func (l *List) LPeek(key string) (*Item, error)

func (*List) LPop 

func (l *List) LPop(key string) (*Record, error)

func (*List) LPush 

func (l *List) LPush(key string, r *Record) error

func (*List) LRange 

func (l *List) LRange(key string, start, end int) ([]*Record, error)

LRange returns the specified elements of the list stored at key [start,end]

func (*List) LRem 

func (l *List) LRem(key string, count int, cmp func(r *Record) (bool, error)) error

LRem removes the first count occurrences of elements equal to value from the list stored at key. The count argument influences the operation in the following ways: count > 0: Remove elements equal to value moving from head to tail. count < 0: Remove elements equal to value moving from tail to head. count = 0: Remove all elements equal to value.

func (*List) LRemByIndex 

func (l *List) LRemByIndex(key string, indexes []int) error

LRemByIndex remove the list element at specified index

func (*List) LTrim 

func (l *List) LTrim(key string, start, end int) error

LTrim trim an existing list so that it will contain only the specified range of elements specified.

func (*List) RPeek 

func (l *List) RPeek(key string) (*Item, error)

func (*List) RPop 

func (l *List) RPop(key string) (*Record, error)

RPop removes and returns the last element of the list stored at key.

func (*List) RPush 

func (l *List) RPush(key string, r *Record) error

func (*List) Size 

func (l *List) Size(key string) (int, error)

type ListIdx 

type ListIdx struct {
	// contains filtered or unexported fields
}

type LruEntry 

type LruEntry struct {
	Key   interface{}
	Value interface{}
}

LruEntry is a struct that represents an entry in the LRU cache.

type MMapRWManager 

type MMapRWManager struct {
	// contains filtered or unexported fields
}

MMapRWManager represents the RWManager which using mmap.

func (*MMapRWManager) Close 

func (mm *MMapRWManager) Close() (err error)

Close will remove the cache in the fdm of the specified path, and call the close method of the os of the file

func (*MMapRWManager) ReadAt 

func (mm *MMapRWManager) ReadAt(b []byte, off int64) (n int, err error)

ReadAt copies data to b slice from mapped region starting at given off and returns number of bytes copied to the b slice.

func (*MMapRWManager) Release 

func (mm *MMapRWManager) Release() (err error)

Release deletes the memory mapped region, flushes any remaining changes

func (*MMapRWManager) Size 

func (mm *MMapRWManager) Size() int64

func (*MMapRWManager) Sync 

func (mm *MMapRWManager) Sync() (err error)

Sync synchronizes the mapping's contents to the file's contents on disk.

func (*MMapRWManager) WriteAt 

func (mm *MMapRWManager) WriteAt(b []byte, off int64) (n int, err error)

WriteAt copies data to mapped region from the b slice starting at given off and returns number of bytes copied to the mapped region.

type MetaData 

type MetaData struct {
	KeySize    uint32
	ValueSize  uint32
	Timestamp  uint64
	TTL        uint32
	Flag       DataFlag // delete / set
	BucketSize uint32
	TxID       uint64
	Status     DataStatus    // committed / uncommitted
	Ds         DataStructure // data structure
	Crc        uint32
	BucketId   BucketId
}

func NewMetaData 

func NewMetaData() *MetaData

func (*MetaData) IsBPlusTree 

func (meta *MetaData) IsBPlusTree() bool

func (*MetaData) IsList 

func (meta *MetaData) IsList() bool

func (*MetaData) IsSet 

func (meta *MetaData) IsSet() bool

func (*MetaData) IsSortSet 

func (meta *MetaData) IsSortSet() bool

func (*MetaData) PayloadSize 

func (meta *MetaData) PayloadSize() int64

func (*MetaData) Size 

func (meta *MetaData) Size() int64

func (*MetaData) WithBucketId 

func (meta *MetaData) WithBucketId(bucketID uint64) *MetaData

func (*MetaData) WithBucketSize 

func (meta *MetaData) WithBucketSize(bucketSize uint32) *MetaData

func (*MetaData) WithCrc 

func (meta *MetaData) WithCrc(crc uint32) *MetaData

func (*MetaData) WithDs 

func (meta *MetaData) WithDs(ds uint16) *MetaData

func (*MetaData) WithFlag 

func (meta *MetaData) WithFlag(flag uint16) *MetaData

func (*MetaData) WithKeySize 

func (meta *MetaData) WithKeySize(keySize uint32) *MetaData

func (*MetaData) WithStatus 

func (meta *MetaData) WithStatus(status uint16) *MetaData

func (*MetaData) WithTTL 

func (meta *MetaData) WithTTL(ttl uint32) *MetaData

func (*MetaData) WithTimeStamp 

func (meta *MetaData) WithTimeStamp(timestamp uint64) *MetaData

func (*MetaData) WithTxID 

func (meta *MetaData) WithTxID(txID uint64) *MetaData

func (*MetaData) WithValueSize 

func (meta *MetaData) WithValueSize(valueSize uint32) *MetaData

type Option 

type Option func(*Options)

func WithBufferSizeOfRecovery 

func WithBufferSizeOfRecovery(size int) Option

func WithCleanFdsCacheThreshold 

func WithCleanFdsCacheThreshold(threshold float64) Option

func WithCommitBufferSize 

func WithCommitBufferSize(commitBufferSize int64) Option

func WithDir 

func WithDir(dir string) Option

func WithEntryIdxMode 

func WithEntryIdxMode(entryIdxMode EntryIdxMode) Option

func WithErrorHandler 

func WithErrorHandler(errorHandler ErrorHandler) Option

func WithGCWhenClose 

func WithGCWhenClose(enable bool) Option

func WithHintKeyAndRAMIdxCacheSize 

func WithHintKeyAndRAMIdxCacheSize(size int) Option

func WithLessFunc 

func WithLessFunc(lessFunc LessFunc) Option

func WithMaxBatchCount 

func WithMaxBatchCount(count int64) Option

func WithMaxBatchSize 

func WithMaxBatchSize(size int64) Option

func WithMaxFdNumsInCache 

func WithMaxFdNumsInCache(num int) Option

func WithMaxWriteRecordCount 

func WithMaxWriteRecordCount(maxWriteRecordCount int64) Option

func WithNodeNum 

func WithNodeNum(num int64) Option

func WithRWMode 

func WithRWMode(rwMode RWMode) Option

func WithSegmentSize 

func WithSegmentSize(size int64) Option

func WithSyncEnable 

func WithSyncEnable(enable bool) Option

type Options 

type Options struct {
	// Dir represents Open the database located in which dir.
	Dir string

	// EntryIdxMode represents using which mode to index the entries.
	EntryIdxMode EntryIdxMode

	// RWMode represents the read and write mode.
	// RWMode includes two options: FileIO and MMap.
	// FileIO represents the read and write mode using standard I/O.
	// MMap represents the read and write mode using mmap.
	RWMode      RWMode
	SegmentSize int64

	// NodeNum represents the node number.
	// Default NodeNum is 1. NodeNum range [1,1023].
	NodeNum int64

	// SyncEnable represents if call Sync() function.
	// if SyncEnable is false, high write performance but potential data loss likely.
	// if SyncEnable is true, slower but persistent.
	SyncEnable bool

	// MaxFdNumsInCache represents the max numbers of fd in cache.
	MaxFdNumsInCache int

	// CleanFdsCacheThreshold represents the maximum threshold for recycling fd, it should be between 0 and 1.
	CleanFdsCacheThreshold float64

	// BufferSizeOfRecovery represents the buffer size of recoveryReader buffer Size
	BufferSizeOfRecovery int

	// CcWhenClose represent initiative GC when calling db.Close()
	GCWhenClose bool

	// CommitBufferSize represent allocated memory for tx
	CommitBufferSize int64

	// ErrorHandler handles an error occurred during transaction.
	// Example:
	//     func triggerAlertError(err error) {
	//     	   if errors.Is(err, targetErr) {
	//         		alertManager.TriggerAlert()
	//     	   }
	//     })
	ErrorHandler ErrorHandler

	// LessFunc is a function that sorts keys.
	LessFunc LessFunc

	// MergeInterval represent the interval for automatic merges, with 0 meaning automatic merging is disabled.
	MergeInterval time.Duration

	// MaxBatchCount represents max entries in batch
	MaxBatchCount int64

	// MaxBatchSize represents max batch size in bytes
	MaxBatchSize int64

	// ExpiredDeleteType represents the data structure used for expired deletion
	// TimeWheel means use the time wheel, You can use it when you need high performance or low memory usage
	// TimeHeap means use the time heap, You can use it when you need to delete precisely or memory usage will be high
	ExpiredDeleteType ExpiredDeleteType

	// max write record num
	MaxWriteRecordCount int64

	// cache size for HintKeyAndRAMIdxMode
	HintKeyAndRAMIdxCacheSize int
}

Options records params for creating DB object.

type RWManager 

type RWManager interface {
	WriteAt(b []byte, off int64) (n int, err error)
	ReadAt(b []byte, off int64) (n int, err error)
	Sync() (err error)
	Release() (err error)
	Size() int64
	Close() (err error)
}

RWManager represents an interface to a RWManager.

type RWMode 

type RWMode int

RWMode represents the read and write mode. 

const (
	// FileIO represents the read and write mode using standard I/O.
	FileIO RWMode = iota

	// MMap represents the read and write mode using mmap.
	MMap
)

type Record 

type Record struct {
	Key       []byte
	Value     []byte
	FileID    int64
	DataPos   uint64
	ValueSize uint32
	Timestamp uint64
	TTL       uint32
	TxID      uint64
}

Record means item of indexes in memory

func NewRecord 

func NewRecord() *Record

NewRecord generate a record Obj

func (*Record) IsExpired 

func (r *Record) IsExpired() bool

IsExpired returns the record if expired or not.

func (*Record) WithDataPos 

func (r *Record) WithDataPos(pos uint64) *Record

WithDataPos set DataPos to Record

func (*Record) WithFileId 

func (r *Record) WithFileId(fid int64) *Record

WithFileId set FileID to Record

func (*Record) WithKey 

func (r *Record) WithKey(k []byte) *Record

func (*Record) WithTTL 

func (r *Record) WithTTL(ttl uint32) *Record

func (*Record) WithTimestamp 

func (r *Record) WithTimestamp(timestamp uint64) *Record

func (*Record) WithTxID 

func (r *Record) WithTxID(txID uint64) *Record

func (*Record) WithValue 

func (r *Record) WithValue(v []byte) *Record

WithValue set the Value to Record

func (*Record) WithValueSize 

func (r *Record) WithValueSize(valueSize uint32) *Record

type SCORE 

type SCORE float64

SCORE represents the score type.

type Set 

type Set struct {
	M map[string]map[uint32]*Record
}

func NewSet 

func NewSet() *Set

func (*Set) SAdd 

func (s *Set) SAdd(key string, values [][]byte, records []*Record) error

SAdd adds the specified members to the set stored at key.

func (*Set) SAreMembers 

func (s *Set) SAreMembers(key string, values ...[]byte) (bool, error)

SAreMembers Returns if members are members of the set stored at key. For multiple items it returns true only if all the items exist.

func (*Set) SCard 

func (s *Set) SCard(key string) int

SCard Returns the set cardinality (number of elements) of the set stored at key.

func (*Set) SDiff 

func (s *Set) SDiff(key1, key2 string) ([]*Record, error)

SDiff Returns the members of the set resulting from the difference between the first set and all the successive sets.

func (*Set) SHasKey 

func (s *Set) SHasKey(key string) bool

SHasKey returns whether it has the set at given key.

func (*Set) SInter 

func (s *Set) SInter(key1, key2 string) ([]*Record, error)

SInter Returns the members of the set resulting from the intersection of all the given sets.

func (*Set) SIsMember 

func (s *Set) SIsMember(key string, value []byte) (bool, error)

SIsMember Returns if member is a member of the set stored at key.

func (*Set) SMembers 

func (s *Set) SMembers(key string) ([]*Record, error)

SMembers returns all the members of the set value stored at key.

func (*Set) SMove 

func (s *Set) SMove(key1, key2 string, value []byte) (bool, error)

SMove moves member from the set at source to the set at destination.

func (*Set) SPop 

func (s *Set) SPop(key string) *Record

SPop removes and returns one or more random elements from the set value store at key.

func (*Set) SRem 

func (s *Set) SRem(key string, values ...[]byte) error

SRem removes the specified members from the set stored at key.

func (*Set) SUnion 

func (s *Set) SUnion(key1, key2 string) ([]*Record, error)

SUnion returns the members of the set resulting from the union of all the given sets.

type SetIdx 

type SetIdx struct {
	// contains filtered or unexported fields
}

type SkipList 

type SkipList struct {
	// contains filtered or unexported fields
}

The SkipList represents the sorted set.

func (*SkipList) FindRank 

func (sl *SkipList) FindRank(hash uint32) int

FindRank Returns the rank of member in the sorted set stored at key, with the scores ordered from low to high. Note that the rank is 1-based integer. Rank 1 means the first node If the node is not found, 0 is returned. Otherwise rank(> 0) is returned.

Time complexity of this method is : O(log(N)).

func (*SkipList) FindRevRank 

func (sl *SkipList) FindRevRank(hash uint32) int

FindRevRank Returns the rank of member in the sorted set stored at key, with the scores ordered from high to low.

func (*SkipList) GetByRank 

func (sl *SkipList) GetByRank(rank int, remove bool) *SkipListNode

GetByRank returns the node at given rank. Note that the rank is 1-based integer. Rank 1 means the first node; Rank -1 means the last node. If remove is true, the returned nodes are removed If node is not found at specific rank, nil is returned.

Time complexity of this method is : O(log(N)).

func (*SkipList) GetByRankRange 

func (sl *SkipList) GetByRankRange(start, end int, remove bool) []*SkipListNode

GetByRankRange returns nodes within specific rank range [start, end]. Note that the rank is 1-based integer. Rank 1 means the first node; Rank -1 means the last node If start is greater than end, the returned array is in reserved order If remove is true, the returned nodes are removed.

Time complexity of this method is : O(log(N)).

func (*SkipList) GetByScoreRange 

func (sl *SkipList) GetByScoreRange(start SCORE, end SCORE, options *GetByScoreRangeOptions) []*SkipListNode

GetByScoreRange returns the nodes whose score within the specific range. If options is nil, it searches in interval [start, end] without any limit by default.

Time complexity of this method is : O(log(N)).

func (*SkipList) GetByValue 

func (sl *SkipList) GetByValue(value []byte) *SkipListNode

GetByValue returns the node at given key. If node is not found, nil is returned

Time complexity : O(1).

func (*SkipList) PeekMax 

func (sl *SkipList) PeekMax() *SkipListNode

PeekMax returns the element with maximum score, nil if the set is empty.

Time Complexity : O(1).

func (*SkipList) PeekMin 

func (sl *SkipList) PeekMin() *SkipListNode

PeekMin returns the element with minimum score, nil if the set is empty.

Time complexity of this method is : O(log(N)).

func (*SkipList) PopMax 

func (sl *SkipList) PopMax() *SkipListNode

PopMax returns and remove the element with maximum score, nil if the set is empty.

Time complexity of this method is : O(log(N)).

func (*SkipList) PopMin 

func (sl *SkipList) PopMin() *SkipListNode

PopMin returns and remove the element with minimal score, nil if the set is empty.

Time complexity of this method is : O(log(N)).

func (*SkipList) Put 

func (sl *SkipList) Put(score SCORE, value []byte, record *Record) error

Put puts an element into the sorted set with specific key / value / score.

Time complexity of this method is : O(log(N)).

func (*SkipList) Remove 

func (sl *SkipList) Remove(hash uint32) *SkipListNode

Remove removes element specified at given key.

Time complexity of this method is : O(log(N)).

func (*SkipList) Size 

func (sl *SkipList) Size() int

Size returns the number of elements in the SkipList.

type SkipListLevel 

type SkipListLevel struct {
	// contains filtered or unexported fields
}

SkipListLevel records forward and span.

type SkipListNode 

type SkipListNode struct {
	// contains filtered or unexported fields
}

SkipListNode represents a node in the SkipList.

func (*SkipListNode) Hash 

func (sln *SkipListNode) Hash() uint32

Hash returns the key of the node.

func (*SkipListNode) Score 

func (sln *SkipListNode) Score() SCORE

Score returns the score of the node.

type SortedSet 

type SortedSet struct {
	M map[string]*SkipList
	// contains filtered or unexported fields
}

func NewSortedSet 

func NewSortedSet(db *DB) *SortedSet

func (*SortedSet) ZAdd 

func (z *SortedSet) ZAdd(key string, score SCORE, value []byte, record *Record) error

func (*SortedSet) ZCard 

func (z *SortedSet) ZCard(key string) (int, error)

func (*SortedSet) ZCount 

func (z *SortedSet) ZCount(key string, start SCORE, end SCORE, opts *GetByScoreRangeOptions) (int, error)

func (*SortedSet) ZExist 

func (z *SortedSet) ZExist(key string, value []byte) (bool, error)

func (*SortedSet) ZMembers 

func (z *SortedSet) ZMembers(key string) (map[*Record]SCORE, error)

func (*SortedSet) ZPeekMax 

func (z *SortedSet) ZPeekMax(key string) (*Record, SCORE, error)

func (*SortedSet) ZPeekMin 

func (z *SortedSet) ZPeekMin(key string) (*Record, SCORE, error)

func (*SortedSet) ZPopMax 

func (z *SortedSet) ZPopMax(key string) (*Record, SCORE, error)

func (*SortedSet) ZPopMin 

func (z *SortedSet) ZPopMin(key string) (*Record, SCORE, error)

func (*SortedSet) ZRangeByRank 

func (z *SortedSet) ZRangeByRank(key string, start int, end int) ([]*Record, []float64, error)

func (*SortedSet) ZRangeByScore 

func (z *SortedSet) ZRangeByScore(key string, start SCORE, end SCORE, opts *GetByScoreRangeOptions) ([]*Record, []float64, error)

func (*SortedSet) ZRank 

func (z *SortedSet) ZRank(key string, value []byte) (int, error)

func (*SortedSet) ZRem 

func (z *SortedSet) ZRem(key string, value []byte) (*Record, error)

func (*SortedSet) ZRemRangeByRank 

func (z *SortedSet) ZRemRangeByRank(key string, start int, end int) error

func (*SortedSet) ZRevRank 

func (z *SortedSet) ZRevRank(key string, value []byte) (int, error)

func (*SortedSet) ZScore 

func (z *SortedSet) ZScore(key string, value []byte) (float64, error)

type SortedSetIdx 

type SortedSetIdx struct {
	// contains filtered or unexported fields
}

type SortedSetMember 

type SortedSetMember struct {
	Value []byte
	Score float64
}

type Throttle 

type Throttle struct {
	// contains filtered or unexported fields
}

Throttle allows a limited number of workers to run at a time. It also provides a mechanism to check for errors encountered by workers and wait for them to finish.

func NewThrottle 

func NewThrottle(max int) *Throttle

NewThrottle creates a new throttle with a max number of workers.

func (*Throttle) Do 

func (t *Throttle) Do() error

Do should be called by workers before they start working. It blocks if there are already maximum number of workers working. If it detects an error from previously Done workers, it would return it.

func (*Throttle) Done 

func (t *Throttle) Done(err error)

Done should be called by workers when they finish working. They can also pass the error status of work done.

func (*Throttle) Finish 

func (t *Throttle) Finish() error

Finish waits until all workers have finished working. It would return any error passed by Done. If Finish is called multiple time, it will wait for workers to finish only once(first time). From next calls, it will return same error as found on first call.

type Tx 

type Tx struct {
	// contains filtered or unexported fields
}

Tx represents a transaction.

func (*Tx) Append 

func (tx *Tx) Append(bucket string, key, appendage []byte) error

func (*Tx) CheckExpire 

func (tx *Tx) CheckExpire(bucket string, key []byte) bool

func (*Tx) Commit 

func (tx *Tx) Commit() (err error)

Commit commits the transaction, following these steps:

1. check the length of pendingWrites.If there are no writes, return immediately.

2. check if the ActiveFile has not enough space to store entry. if not, call rotateActiveFile function.

3. write pendingWrites to disk, if a non-nil error,return the error.

4. build Hint index.

5. Unlock the database and clear the db field.

func (*Tx) CommitWith 

func (tx *Tx) CommitWith(cb func(error))

func (*Tx) Decr 

func (tx *Tx) Decr(bucket string, key []byte) error

func (*Tx) DecrBy 

func (tx *Tx) DecrBy(bucket string, key []byte, decrement int64) error

func (*Tx) Delete 

func (tx *Tx) Delete(bucket string, key []byte) error

Delete removes a key from the bucket at given bucket and key.

func (*Tx) DeleteBucket 

func (tx *Tx) DeleteBucket(ds uint16, bucket string) error

DeleteBucket delete bucket depends on ds (represents the data structure)

func (*Tx) ExistBucket 

func (tx *Tx) ExistBucket(ds uint16, bucket string) bool

func (*Tx) ExpireList 

func (tx *Tx) ExpireList(bucket string, key []byte, ttl uint32) error

func (*Tx) Get 

func (tx *Tx) Get(bucket string, key []byte) (value []byte, err error)

Get retrieves the value for a key in the bucket. The returned value is only valid for the life of the transaction.

func (*Tx) GetAll 

func (tx *Tx) GetAll(bucket string) ([][]byte, [][]byte, error)

GetAll returns all keys and values of the bucket stored at given bucket.

func (*Tx) GetBit 

func (tx *Tx) GetBit(bucket string, key []byte, offset int) (byte, error)

func (*Tx) GetKeys 

func (tx *Tx) GetKeys(bucket string) ([][]byte, error)

GetKeys returns all keys of the bucket stored at given bucket.

func (*Tx) GetListTTL 

func (tx *Tx) GetListTTL(bucket string, key []byte) (uint32, error)

func (*Tx) GetMaxKey 

func (tx *Tx) GetMaxKey(bucket string) ([]byte, error)

func (*Tx) GetMinKey 

func (tx *Tx) GetMinKey(bucket string) ([]byte, error)

func (*Tx) GetRange 

func (tx *Tx) GetRange(bucket string, key []byte, start, end int) ([]byte, error)

func (*Tx) GetSet 

func (tx *Tx) GetSet(bucket string, key, value []byte) (oldValue []byte, err error)

func (*Tx) GetTTL 

func (tx *Tx) GetTTL(bucket string, key []byte) (int64, error)

GetTTL returns remaining TTL of a value by key. It returns (-1, nil) If TTL is Persistent (0, ErrBucketNotFound|ErrKeyNotFound) If expired or not found (TTL, nil) If the record exists with a TTL Note: The returned remaining TTL will be in seconds. For example, remainingTTL is 500ms, It'll return 0.

func (*Tx) GetValues 

func (tx *Tx) GetValues(bucket string) ([][]byte, error)

GetValues returns all values of the bucket stored at given bucket.

func (*Tx) Incr 

func (tx *Tx) Incr(bucket string, key []byte) error

func (*Tx) IncrBy 

func (tx *Tx) IncrBy(bucket string, key []byte, increment int64) error

func (*Tx) IterateBuckets 

func (tx *Tx) IterateBuckets(ds uint16, pattern string, f func(bucket string) bool) error

IterateBuckets iterate over all the bucket depends on ds (represents the data structure)

func (*Tx) LKeys 

func (tx *Tx) LKeys(bucket, pattern string, f func(key string) bool) error

LKeys find all keys matching a given pattern

func (*Tx) LKeysAll 

func (tx *Tx) LKeysAll(bucket string, f func(key string) bool) error

LKeysAll find all keys without matching any pattern

func (*Tx) LKeysRaw 

func (tx *Tx) LKeysRaw(bucket string) ([]string, error)

LKeysRaw find all keys as string array

func (*Tx) LPeek 

func (tx *Tx) LPeek(bucket string, key []byte) (item []byte, err error)

LPeek returns the first element of the list stored in the bucket at given bucket and key.

func (*Tx) LPop 

func (tx *Tx) LPop(bucket string, key []byte) (item []byte, err error)

LPop removes and returns the first element of the list stored in the bucket at given bucket and key.

func (*Tx) LPush 

func (tx *Tx) LPush(bucket string, key []byte, values ...[]byte) error

LPush inserts the values at the head of the list stored in the bucket at given bucket,key and values.

func (*Tx) LPushRaw 

func (tx *Tx) LPushRaw(bucket string, key []byte, values ...[]byte) error

func (*Tx) LRange 

func (tx *Tx) LRange(bucket string, key []byte, start, end int) ([][]byte, error)

LRange returns the specified elements of the list stored in the bucket at given bucket,key, start and end. The offsets start and stop are zero-based indexes 0 being the first element of the list (the head of the list), 1 being the next element and so on. Start and end can also be negative numbers indicating offsets from the end of the list, where -1 is the last element of the list, -2 the penultimate element and so on.

func (*Tx) LRem 

func (tx *Tx) LRem(bucket string, key []byte, count int, value []byte) error

LRem removes the first count occurrences of elements equal to value from the list stored in the bucket at given bucket,key,count. The count argument influences the operation in the following ways: count > 0: Remove elements equal to value moving from head to tail. count < 0: Remove elements equal to value moving from tail to head. count = 0: Remove all elements equal to value.

func (*Tx) LRemByIndex 

func (tx *Tx) LRemByIndex(bucket string, key []byte, indexes ...int) error

LRemByIndex remove the list element at specified index

func (*Tx) LSize 

func (tx *Tx) LSize(bucket string, key []byte) (int, error)

LSize returns the size of key in the bucket in the bucket at given bucket and key.

func (*Tx) LTrim 

func (tx *Tx) LTrim(bucket string, key []byte, start, end int) error

LTrim trims an existing list so that it will contain only the specified range of elements specified. the offsets start and stop are zero-based indexes 0 being the first element of the list (the head of the list), 1 being the next element and so on. start and end can also be negative numbers indicating offsets from the end of the list, where -1 is the last element of the list, -2 the penultimate element and so on.

func (*Tx) MGet 

func (tx *Tx) MGet(bucket string, keys ...[]byte) ([][]byte, error)

func (*Tx) MSet 

func (tx *Tx) MSet(bucket string, ttl uint32, args ...[]byte) error

func (*Tx) NewBucket 

func (tx *Tx) NewBucket(ds uint16, name string) (err error)

func (*Tx) NewKVBucket 

func (tx *Tx) NewKVBucket(name string) error

func (*Tx) NewListBucket 

func (tx *Tx) NewListBucket(name string) error

func (*Tx) NewSetBucket 

func (tx *Tx) NewSetBucket(name string) error

func (*Tx) NewSortSetBucket 

func (tx *Tx) NewSortSetBucket(name string) error

func (*Tx) Persist 

func (tx *Tx) Persist(bucket string, key []byte) error

Persist updates record's TTL as Persistent if the record exits.

func (*Tx) PrefixScan 

func (tx *Tx) PrefixScan(bucket string, prefix []byte, offsetNum int, limitNum int) (values [][]byte, err error)

PrefixScan iterates over a key prefix at given bucket, prefix and limitNum. LimitNum will limit the number of entries return.

func (*Tx) PrefixSearchScan 

func (tx *Tx) PrefixSearchScan(bucket string, prefix []byte, reg string, offsetNum int, limitNum int) (values [][]byte, err error)

PrefixSearchScan iterates over a key prefix at given bucket, prefix, match regular expression and limitNum. LimitNum will limit the number of entries return.

func (*Tx) Put 

func (tx *Tx) Put(bucket string, key, value []byte, ttl uint32) error

Put sets the value for a key in the bucket. a wrapper of the function put.

func (*Tx) PutIfExists 

func (tx *Tx) PutIfExists(bucket string, key, value []byte, ttl uint32) error

PutIfExits set the value for a key in the bucket only if the key already exits.

func (*Tx) PutIfNotExists 

func (tx *Tx) PutIfNotExists(bucket string, key, value []byte, ttl uint32) error

PutIfNotExists set the value for a key in the bucket only if the key doesn't exist already.

func (*Tx) PutWithTimestamp 

func (tx *Tx) PutWithTimestamp(bucket string, key, value []byte, ttl uint32, timestamp uint64) error

func (*Tx) RPeek 

func (tx *Tx) RPeek(bucket string, key []byte) ([]byte, error)

RPeek returns the last element of the list stored in the bucket at given bucket and key.

func (*Tx) RPop 

func (tx *Tx) RPop(bucket string, key []byte) (item []byte, err error)

RPop removes and returns the last element of the list stored in the bucket at given bucket and key.

func (*Tx) RPush 

func (tx *Tx) RPush(bucket string, key []byte, values ...[]byte) error

RPush inserts the values at the tail of the list stored in the bucket at given bucket,key and values.

func (*Tx) RPushRaw 

func (tx *Tx) RPushRaw(bucket string, key []byte, values ...[]byte) error

func (*Tx) RangeScan 

func (tx *Tx) RangeScan(bucket string, start, end []byte) (values [][]byte, err error)

RangeScan query a range at given bucket, start and end slice.

func (*Tx) Rollback 

func (tx *Tx) Rollback() error

Rollback closes the transaction.

func (*Tx) SAdd 

func (tx *Tx) SAdd(bucket string, key []byte, items ...[]byte) error

SAdd adds the specified members to the set stored int the bucket at given bucket,key and items.

func (*Tx) SAreMembers 

func (tx *Tx) SAreMembers(bucket string, key []byte, items ...[]byte) (bool, error)

SAreMembers returns if the specified members are the member of the set int the bucket at given bucket,key and items.

func (*Tx) SCard 

func (tx *Tx) SCard(bucket string, key []byte) (int, error)

SCard returns the set cardinality (number of elements) of the set stored in the bucket at given bucket and key.

func (*Tx) SDiffByOneBucket 

func (tx *Tx) SDiffByOneBucket(bucket string, key1, key2 []byte) ([][]byte, error)

SDiffByOneBucket returns the members of the set resulting from the difference between the first set and all the successive sets in one bucket.

func (*Tx) SDiffByTwoBuckets 

func (tx *Tx) SDiffByTwoBuckets(bucket1 string, key1 []byte, bucket2 string, key2 []byte) ([][]byte, error)

SDiffByTwoBuckets returns the members of the set resulting from the difference between the first set and all the successive sets in two buckets.

func (*Tx) SHasKey 

func (tx *Tx) SHasKey(bucket string, key []byte) (bool, error)

SHasKey returns if the set in the bucket at given bucket and key.

func (*Tx) SIsMember 

func (tx *Tx) SIsMember(bucket string, key, item []byte) (bool, error)

SIsMember returns if member is a member of the set stored int the bucket at given bucket,key and item.

func (*Tx) SKeys 

func (tx *Tx) SKeys(bucket, pattern string, f func(key string) bool) error

SKeys find all keys matching a given pattern

func (*Tx) SMembers 

func (tx *Tx) SMembers(bucket string, key []byte) ([][]byte, error)

SMembers returns all the members of the set value stored int the bucket at given bucket and key.

func (*Tx) SMoveByOneBucket 

func (tx *Tx) SMoveByOneBucket(bucket string, key1, key2, item []byte) (bool, error)

SMoveByOneBucket moves member from the set at source to the set at destination in one bucket.

func (*Tx) SMoveByTwoBuckets 

func (tx *Tx) SMoveByTwoBuckets(bucket1 string, key1 []byte, bucket2 string, key2, item []byte) (bool, error)

SMoveByTwoBuckets moves member from the set at source to the set at destination in two buckets.

func (*Tx) SPop 

func (tx *Tx) SPop(bucket string, key []byte) ([]byte, error)

SPop removes and returns one or more random elements from the set value store in the bucket at given bucket and key.

func (*Tx) SRem 

func (tx *Tx) SRem(bucket string, key []byte, items ...[]byte) error

SRem removes the specified members from the set stored int the bucket at given bucket,key and items.

func (*Tx) SUnionByOneBucket 

func (tx *Tx) SUnionByOneBucket(bucket string, key1, key2 []byte) ([][]byte, error)

SUnionByOneBucket the members of the set resulting from the union of all the given sets in one bucket.

func (*Tx) SUnionByTwoBuckets 

func (tx *Tx) SUnionByTwoBuckets(bucket1 string, key1 []byte, bucket2 string, key2 []byte) ([][]byte, error)

SUnionByTwoBuckets the members of the set resulting from the union of all the given sets in two buckets.

func (*Tx) SetBit 

func (tx *Tx) SetBit(bucket string, key []byte, offset int, bit byte) error

func (*Tx) SubmitBucket 

func (tx *Tx) SubmitBucket() error

func (*Tx) ValueLen 

func (tx *Tx) ValueLen(bucket string, key []byte) (int, error)

func (*Tx) ZAdd 

func (tx *Tx) ZAdd(bucket string, key []byte, score float64, val []byte) error

ZAdd Adds the specified member with the specified score into the sorted set specified by key in a bucket.

func (*Tx) ZCard 

func (tx *Tx) ZCard(bucket string, key []byte) (int, error)

ZCard Returns the sorted set cardinality (number of elements) of the sorted set specified by key in a bucket.

func (*Tx) ZCheck 

func (tx *Tx) ZCheck(bucket string) error

func (*Tx) ZCount 

func (tx *Tx) ZCount(bucket string, key []byte, start, end float64, opts *GetByScoreRangeOptions) (int, error)

ZCount Returns the number of elements in the sorted set specified by key in a bucket with a score between min and max and opts. Opt includes the following parameters: Limit int // limit the max nodes to return ExcludeStart bool // exclude start value, so it search in interval (start, end] or (start, end) ExcludeEnd bool // exclude end value, so it search in interval [start, end) or (start, end)

func (*Tx) ZKeys 

func (tx *Tx) ZKeys(bucket, pattern string, f func(key string) bool) error

ZKeys find all keys matching a given pattern in a bucket

func (*Tx) ZMembers 

func (tx *Tx) ZMembers(bucket string, key []byte) (map[*SortedSetMember]struct{}, error)

ZMembers Returns all the members and scores of members of the set specified by key in a bucket.

func (*Tx) ZPeekMax 

func (tx *Tx) ZPeekMax(bucket string, key []byte) (*SortedSetMember, error)

ZPeekMax Returns the member with the highest score in the sorted set specified by key in a bucket.

func (*Tx) ZPeekMin 

func (tx *Tx) ZPeekMin(bucket string, key []byte) (*SortedSetMember, error)

ZPeekMin Returns the member with the lowest score in the sorted set specified by key in a bucket.

func (*Tx) ZPopMax 

func (tx *Tx) ZPopMax(bucket string, key []byte) (*SortedSetMember, error)

ZPopMax Removes and returns the member with the highest score in the sorted set specified by key in a bucket.

func (*Tx) ZPopMin 

func (tx *Tx) ZPopMin(bucket string, key []byte) (*SortedSetMember, error)

ZPopMin Removes and returns the member with the lowest score in the sorted set specified by key in a bucket.

func (*Tx) ZRangeByRank 

func (tx *Tx) ZRangeByRank(bucket string, key []byte, start, end int) ([]*SortedSetMember, error)

ZRangeByRank Returns all the elements in the sorted set specified by key in a bucket with a rank between start and end (including elements with rank equal to start or end).

func (*Tx) ZRangeByScore 

func (tx *Tx) ZRangeByScore(bucket string, key []byte, start, end float64, opts *GetByScoreRangeOptions) ([]*SortedSetMember, error)

ZRangeByScore Returns all the elements in the sorted set specified by key in a bucket with a score between min and max. And the parameter `Opts` is the same as ZCount's.

func (*Tx) ZRank 

func (tx *Tx) ZRank(bucket string, key, value []byte) (int, error)

ZRank Returns the rank of member in the sorted set specified by key in a bucket, with the scores ordered from low to high.

func (*Tx) ZRem 

func (tx *Tx) ZRem(bucket string, key []byte, value []byte) error

ZRem removes the specified members from the sorted set stored in one bucket at given bucket and key.

func (*Tx) ZRemRangeByRank 

func (tx *Tx) ZRemRangeByRank(bucket string, key []byte, start, end int) error

ZRemRangeByRank removes all elements in the sorted set stored in one bucket at given bucket with rank between start and end. the rank is 1-based integer. Rank 1 means the first node; Rank -1 means the last node.

func (*Tx) ZRevRank 

func (tx *Tx) ZRevRank(bucket string, key, value []byte) (int, error)

ZRevRank Returns the rank of member in the sorted set specified by key in a bucket, with the scores ordered from high to low.

func (*Tx) ZScore 

func (tx *Tx) ZScore(bucket string, key, value []byte) (float64, error)

ZScore Returns the score of members in a sorted set specified by key in a bucket.

type WriteBatch 

type WriteBatch struct {
	sync.Mutex
	// contains filtered or unexported fields
}

WriteBatch holds the necessary info to perform batched writes.

func (*WriteBatch) Cancel 

func (wb *WriteBatch) Cancel() error

func (*WriteBatch) Delete 

func (wb *WriteBatch) Delete(bucket string, key []byte) error

func (tx *Tx) Delete(bucket string, key []byte) error

func (*WriteBatch) Error 

func (wb *WriteBatch) Error() error

Error returns any errors encountered so far. No commits would be run once an error is detected.

func (*WriteBatch) Flush 

func (wb *WriteBatch) Flush() error

func (*WriteBatch) Put 

func (wb *WriteBatch) Put(bucket string, key, value []byte, ttl uint32) error

func (*WriteBatch) Reset 

func (wb *WriteBatch) Reset() error

func (*WriteBatch) SetMaxPendingTxns 

func (wb *WriteBatch) SetMaxPendingTxns(max int)

SetMaxPendingTxns sets a limit on maximum number of pending transactions while writing batches. This function should be called before using WriteBatch. Default value of MaxPendingTxns is 16 to minimise memory usage.

Source Files

View all Source files

Directories

Path Synopsis
examples
basic command
batch/concurrence command
batch/put command
batch/read command
bucket command
http command
iterator command
k-v/append command
k-v/bit command
k-v/get_range command
k-v/increments command
k-v/multioprations command
list command
set command
sortedSet command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.