metabase

package
v1.148.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 2, 2026 License: AGPL-3.0 Imports: 55 Imported by: 10

Documentation

Overview

Package metabase implements storing objects and segements.

Package metabase provides the core metadata storage layer for Storj, managing all information about objects (files) and segments (pieces of files) stored in the network.

Architecture Overview

Metabase acts as the single source of truth for what data exists in the Storj network. It supports multiple database backends (PostgreSQL, CockroachDB, Google Cloud Spanner) through an adapter pattern, allowing the same business logic to work across different database implementations.

Key Components:

  • DB: Main database handle with connection management (db.go)
  • Adapter: Interface abstracting database operations (adapter.go)
  • NodeAliasCache: Maps NodeIDs to compact aliases (aliascache.go)

Core Data Model

The metabase stores two primary entity types:

Objects (RawObject in raw.go):

  • Represents a complete file in the network
  • Key fields: ProjectID, BucketName, ObjectKey, Version
  • Status: Pending, CommittedUnversioned, CommittedVersioned, DeleteMarkerVersioned, DeleteMarkerUnversioned
  • Contains metadata: size, encryption, expiration, retention
  • ZombieDeletionDeadline prevents abandoned uploads from bloating database

Segments (RawSegment in raw.go):

  • Represents (usually up to a 64MB but not always) piece of an object
  • Contains piece locations (which storage nodes hold erasure-coded data)
  • Position encoded as uint64: upper 32 bits = Part, lower 32 bits = Index
  • Can be inline (small data stored directly) or remote (references to storage nodes)

Constraints:

  • It should not be possible to modify committed objects and delete markers (except for Object Lock metadata with proper authorization).
  • It should not be possible to modify committed object segments.
  • When committing a versioned object or delete marker with auto-assigned version, the new object should have the largest committed version.
  • It should not be possible to modify or delete a locked object.
  • An object and its segments should be atomically committed.
  • There should only be one committed unversioned object or delete marker per object location (project_id, bucket_name, object_key).
  • Segment positions must be unique within an object.
  • StreamID must be unique across the system.

Database Schema

Main tables:

  • objects: Primary key (project_id, bucket_name, object_key, version)
  • segments: Primary key (stream_id, position), foreign key to objects
  • node_aliases: Maps NodeID (32 bytes) to NodeAlias (4 bytes) for space efficiency

Migrations are managed per backend:

  • PostgreSQL/CockroachDB: PostgresMigration() (db.go)
  • Spanner: SpannerMigration() (db.go)

Supported Operations

Object Lifecycle:

  • BeginObjectNextVersion: Start new upload with auto-assigned version (commit.go)
  • BeginObjectExactVersion: Start upload with specific version
  • CommitObject: Finalize upload, transition from Pending to Committed
  • GetObjectExactVersion: Fetch specific version (get.go)
  • GetObjectLastCommitted: Fetch latest non-pending version (get.go)
  • DeleteObjectExactVersion: Delete specific version with Object Lock support (delete.go)
  • ListObjects: Iterate objects with optional prefix/delimiter (iterator.go)

Segment Operations:

  • BeginSegment: Reserve segment position
  • CommitSegment: Write segment metadata with piece locations
  • IterateLoopSegments: Efficiently scan all segments for background jobs (loop.go)

Database Backends

PostgreSQL:

  • Default development backend
  • Full feature support
  • Connection string: "postgres://..."

CockroachDB:

  • Production backend for distributed deployments
  • Optimizations for CRDB-specific features
  • Connection string: "cockroach://..."

Spanner:

  • Google Cloud Spanner for global distribution
  • Supports change streams for CDC (see changestream package)
  • Some DML limitations on primary keys
  • Connection string: "spanner://..."

Multi-Adapter Support

The DB can manage multiple adapters for different projects:

  • ChooseAdapter(projectID) selects the appropriate backend (db.go)
  • Useful for gradual migrations between database types
  • Node alias coordination across adapters (first adapter is source of truth)

Node Aliases

To reduce storage requirements, metabase uses 4-byte NodeAlias instead of 32-byte NodeID in the segments table:

  • NodeAlias: int32 type (alias.go)
  • NodeAliasCache: Write-through cache (aliascache.go)
  • EnsureNodeAliases: Create aliases if they don't exist (alias.go)
  • AliasPieces: Compressed piece representation (aliaspiece.go)

This optimization significantly reduces segments table size in production.

Object Versioning

Objects support versioning similar to S3:

  • Version 0 (NextVersion): Auto-assigned next version
  • Version >0: Explicit version number
  • StreamVersionID: Public API combining Version (8 bytes) + StreamID suffix (8 bytes)
  • Versioned buckets can have multiple versions per key
  • Unversioned buckets have single version, new uploads replace old

Delete markers (DeleteMarkerVersioned, DeleteMarkerUnversioned) represent soft deletions.

Object Lock and Retention

Supports S3 Object Lock features:

  • Retention: Time-based protection (governance or compliance mode)
  • Legal Hold: Indefinite protection flag
  • Governance mode can be bypassed with special permission
  • Compliance mode cannot be bypassed
  • Validated in BeginObjectNextVersion (commit.go) and DeleteObjectExactVersion (delete.go)

Zombie Objects

Pending objects that are never committed become "zombies":

  • ZombieDeletionDeadline set at object creation (default 24 hours)
  • Background cleanup handled by zombiedeletion package
  • Prevents database bloat from abandoned uploads

Integration with Satellite

Other satellite components use metabase for:

  • Metainfo service: Upload/download metadata operations
  • Repair service: Find under-replicated segments via rangedloop
  • Audit service: Select segments for verification
  • Garbage collection: Generate bloom filters of retained pieces
  • Accounting: Tally stored data for billing

Common Patterns

Error Handling:

  • Uses github.com/zeebo/errs for error classes
  • Common errors: ErrObjectNotFound, ErrInvalidRequest, ErrConflict
  • All errors wrapped with context

Monitoring:

  • All functions use defer mon.Task()(&ctx)(&err) for metrics

Verification:

  • Request structs have Verify() method validating fields

Context Propagation:

  • All functions take context.Context as first parameter

Testing

Use metabasetest package for testing:

  • Declarative test steps with Check() methods
  • Helper functions: RandObjectStream(), CreateObject(), etc.
  • TestingGetState() verifies complete database state (raw.go)
  • TestingBatchInsertObjects/Segments for bulk test data (raw.go)

Making Changes

When modifying metabase code:

  1. Test against all adapters (PostgreSQL, CockroachDB, Spanner)
  2. Write reversible migrations
  3. Consider query performance (metabase is frequently queried)
  4. Maintain versioning logic carefully
  5. Enforce Object Lock constraints
  6. Always use node aliases in segments, never raw NodeIDs
  7. Set ZombieDeletionDeadline when creating pending objects
  8. Filter out pending and expired objects in queries

Key Files Reference

  • db.go: DB type, Open(), adapter management
  • adapter.go: Adapter interface, PostgresAdapter, CockroachAdapter
  • common.go: Core types (ObjectLocation, ObjectStream, SegmentPosition, etc.)
  • raw.go: RawObject, RawSegment, testing utilities
  • commit.go: BeginObject, CommitObject operations
  • get.go: GetObject operations
  • delete.go: DeleteObject operations
  • iterator.go: ListObjects implementation
  • loop.go: IterateLoopSegments for background processing
  • alias.go: Node alias operations
  • aliascache.go: NodeAliasCache implementation
  • metabase/zombiedeletion: Automatic cleanup of abandoned uploads
  • metabase/rangedloop: Parallel segment processing framework
  • metabase/avrometabase: Parse segments from Avro files
  • metabase/changestream: Spanner change data capture
  • metabase/metabasetest: Testing utilities

Index

Constants

View Source 
const (
	Delimiter = "/"
	// DelimiterNext is the string that comes immediately after Delimiter="/".
	DelimiterNext    = "0"
	LastSegmentName  = "l"
	LastSegmentIndex = uint32(math.MaxUint32)
)

Common constants for segment keys.

View Source 
const (
	// Pending means that the object is being uploaded or that the client failed during upload.
	// The failed upload may be continued in the future.
	Pending = ObjectStatus(1)

	// CommittedUnversioned means that the object is finished and should be visible for general listing.
	CommittedUnversioned = ObjectStatus(3)
	// CommittedVersioned means that the object is finished and should be visible for general listing.
	CommittedVersioned = ObjectStatus(4)
	// DeleteMarkerVersioned is inserted when an object is deleted in a versioning enabled bucket.
	DeleteMarkerVersioned = ObjectStatus(5)
	// DeleteMarkerUnversioned is inserted when an unversioned object is deleted in a versioning suspended bucket.
	DeleteMarkerUnversioned = ObjectStatus(6)

	// Prefix is an ephemeral status used during non-recursive listing.
	Prefix = ObjectStatus(7)

	// DefaultStatus is the default status for new objects.
	DefaultStatus = Pending
)
View Source 
const DefaultVersion = Version(1)

DefaultVersion represents default version 1.

View Source 
const DeleteObjectsMaxItems = 1000

DeleteObjectsMaxItems is the maximum amount of items that are allowed in a DeleteObjects request.

View Source 
const ListLimit = intLimitRange(1000)

ListLimit is the maximum number of items the client can request for listing.

View Source 
const ListVerifyLimit = intLimitRange(100000)

ListVerifyLimit is the maximum number of items the client can request for listing.

View Source 
const MaxVersion = Version(math.MaxInt64 - 64)

MaxVersion represents maximum version. Version in DB is represented as INT8.

It uses `MaxInt64 - 64` to avoid issues with `-MaxVersion`.

View Source 
const NextVersion = Version(0)

NextVersion means that the version should be chosen automatically.

View Source 
const ValidatePlainSize = false

ValidatePlainSize determines whether we disable PlainSize validation for old uplinks.

Variables

View Source 
var (
	// ErrObjectNotFound is used to indicate that the object does not exist.
	ErrObjectNotFound = errs.Class("object not found")
	// ErrInvalidRequest is used to indicate invalid requests.
	ErrInvalidRequest = errs.Class("metabase: invalid request")
	// ErrFailedPrecondition is used to indicate that some conditions in the request has failed.
	ErrFailedPrecondition = errs.Class("metabase: failed precondition")
	// ErrConflict is used to indicate conflict with the request.
	ErrConflict = errs.Class("metabase: conflict")
)
View Source 
var (
	// Error is the default error for metabase.
	Error = errs.Class("metabase")
	// ErrObjectAlreadyExists is used to indicate that object already exists.
	ErrObjectAlreadyExists = errs.Class("object already exists")
	// ErrPendingObjectMissing is used to indicate a pending object is no longer accessible.
	ErrPendingObjectMissing = errs.Class("pending object missing")
	// ErrPermissionDenied general error for denying permission.
	ErrPermissionDenied = errs.Class("permission denied")
	// ErrMethodNotAllowed general error when operation is not allowed.
	ErrMethodNotAllowed = errs.Class("method not allowed")
	// ErrUnimplemented is used to indicate an option was not implemented.
	ErrUnimplemented = errs.Class("not implemented")
)
View Source 
var (
	// ErrValueChanged is returned when the current value of the key does not match the oldValue in UpdateSegmentPieces.
	ErrValueChanged = errs.Class("value changed")
	// ErrObjectExpiration is used when an object's expiration prevents an operation from succeeding.
	ErrObjectExpiration = errs.Class("object expiration")
	// ErrObjectStatus is used when an object's status prevents an operation from succeeding.
	ErrObjectStatus = errs.Class("object status")
)
View Source 
var ErrObjectLock = errs.Class("object lock")

ErrObjectLock is used when an object's Object Lock configuration prevents an operation from succeeding.

View Source 
var ErrSegmentNotFound = errs.Class("segment not found")

ErrSegmentNotFound is an error class for non-existing segment.

Functions

func EqualAliasPieces 

func EqualAliasPieces(xs, ys AliasPieces) bool

EqualAliasPieces compares whether xs and ys are equal.

func IsFinalPrefix added in v1.133.2 

func IsFinalPrefix(prefix ObjectKey) bool

IsFinalPrefix returns true when the prefix has no object keys after.

func LessObjectKey added in v1.106.1 

func LessObjectKey(a, b ObjectKey) bool

LessObjectKey returns whether a < b.

func LimitedAsOfSystemTime added in v1.106.1 

func LimitedAsOfSystemTime(impl dbutil.Implementation, now, baseline time.Time, maxInterval time.Duration) string

LimitedAsOfSystemTime returns a SQL query clause for AS OF SYSTEM TIME.

func MigrateMetainfoDB added in v1.120.1 

func MigrateMetainfoDB(ctx context.Context, log *zap.Logger, db *DB, migrationType string) (err error)

MigrateMetainfoDB migrates metabase database.

func Module added in v1.120.1 

func Module(ball *mud.Ball)

Module is a mud module.

func SpannerTestModule added in v1.103.2 

func SpannerTestModule(ball *mud.Ball, spannerConnection string)

SpannerTestModule adds all the required dependencies for Spanner migration and adapter.

func TrimAfterDelimiter added in v1.133.2 

func TrimAfterDelimiter(s string, delimiter string) (trimmed string, ok bool)

TrimAfterDelimiter removes the portion of the string that follows the first instance of the delimiter. If the delimiter was not found, ok will be false and the string will be returned unchanged.

Types

type Adapter added in v1.102.2 

type Adapter interface {
	Shard

	Name() string
	Now(ctx context.Context) (time.Time, error)
	Ping(ctx context.Context) error
	MigrateToLatest(ctx context.Context) error
	CheckVersion(ctx context.Context) error
	Implementation() dbutil.Implementation

	BeginObjectNextVersion(context.Context, BeginObjectNextVersion, *Object) error
	GetObjectLastCommitted(ctx context.Context, opts GetObjectLastCommitted) (Object, error)
	IterateLoopSegments(ctx context.Context, aliasCache *NodeAliasCache, opts IterateLoopSegments, fn func(context.Context, LoopSegmentsIterator) error) error
	PendingObjectExists(ctx context.Context, opts BeginSegment) (exists bool, err error)
	CommitPendingObjectSegment(ctx context.Context, opts CommitSegment, aliasPieces AliasPieces) error
	CommitInlineSegment(ctx context.Context, opts CommitInlineSegment) error
	BeginObjectExactVersion(ctx context.Context, opts BeginObjectExactVersion, object *Object) error

	GetObjectExactVersionRetention(ctx context.Context, opts GetObjectExactVersionRetention) (retention Retention, err error)
	GetObjectLastCommittedRetention(ctx context.Context, opts GetObjectLastCommittedRetention) (retention Retention, err error)
	SetObjectExactVersionRetention(ctx context.Context, opts SetObjectExactVersionRetention) error
	SetObjectLastCommittedRetention(ctx context.Context, opts SetObjectLastCommittedRetention) error

	GetObjectExactVersionLegalHold(ctx context.Context, opts GetObjectExactVersionLegalHold) (enabled bool, err error)
	GetObjectLastCommittedLegalHold(ctx context.Context, opts GetObjectLastCommittedLegalHold) (enabled bool, err error)
	SetObjectExactVersionLegalHold(ctx context.Context, opts SetObjectExactVersionLegalHold) error
	SetObjectLastCommittedLegalHold(ctx context.Context, opts SetObjectLastCommittedLegalHold) error

	GetTableStats(ctx context.Context, opts GetTableStats) (result TableStats, err error)
	CountSegments(ctx context.Context, checkTimestamp time.Time) (result int64, err error)
	UpdateTableStats(ctx context.Context) error
	BucketEmpty(ctx context.Context, opts BucketEmpty) (empty bool, err error)

	WithTx(ctx context.Context, opts TransactionOptions, f func(context.Context, TransactionAdapter) error) error

	CollectBucketTallies(ctx context.Context, opts CollectBucketTallies) (result []BucketTally, err error)

	GetSegmentByPosition(ctx context.Context, opts GetSegmentByPosition) (segment Segment, aliasPieces AliasPieces, err error)
	GetSegmentByPositionForAudit(ctx context.Context, opts GetSegmentByPosition) (segment SegmentForAudit, aliasPieces AliasPieces, err error)
	GetSegmentByPositionForRepair(ctx context.Context, opts GetSegmentByPosition) (segment SegmentForRepair, aliasPieces AliasPieces, err error)
	CheckSegmentPiecesAlteration(ctx context.Context, streamID uuid.UUID, position SegmentPosition, aliasPieces AliasPieces) (altered bool, err error)
	GetObjectExactVersion(ctx context.Context, opts GetObjectExactVersion) (_ Object, err error)
	GetSegmentPositionsAndKeys(ctx context.Context, streamID uuid.UUID) (keysNonces []EncryptedKeyAndNonce, err error)
	GetLatestObjectLastSegment(ctx context.Context, opts GetLatestObjectLastSegment) (segment Segment, aliasPieces AliasPieces, err error)

	ListObjects(ctx context.Context, opts ListObjects) (result ListObjectsResult, err error)
	ListSegments(ctx context.Context, opts ListSegments, aliasCache *NodeAliasCache) (result ListSegmentsResult, err error)
	ListStreamPositions(ctx context.Context, opts ListStreamPositions) (result ListStreamPositionsResult, err error)
	ListVerifySegments(ctx context.Context, opts ListVerifySegments) (segments []VerifySegment, err error)
	ListBucketStreamIDs(ctx context.Context, opts ListBucketStreamIDs, process func(ctx context.Context, streamIDs []uuid.UUID) error) (err error)

	UpdateSegmentPieces(ctx context.Context, opts UpdateSegmentPieces, oldPieces, newPieces AliasPieces) (resultPieces AliasPieces, err error)
	UpdateObjectLastCommittedMetadata(ctx context.Context, opts UpdateObjectLastCommittedMetadata) (affected int64, err error)

	DeleteObjectExactVersion(ctx context.Context, opts DeleteObjectExactVersion) (result DeleteObjectResult, err error)
	DeletePendingObject(ctx context.Context, opts DeletePendingObject) (result DeleteObjectResult, err error)

	DeleteObjectLastCommittedPlain(ctx context.Context, opts DeleteObjectLastCommitted) (result DeleteObjectResult, err error)
	DeleteObjectLastCommittedVersioned(ctx context.Context, opts DeleteObjectLastCommitted, deleterMarkerStreamID uuid.UUID) (result DeleteObjectResult, err error)

	IterateExpiredObjects(ctx context.Context, opts DeleteExpiredObjects, process func(context.Context, []ObjectStream) error) (err error)
	DeleteObjectsAndSegmentsNoVerify(ctx context.Context, objects []ObjectStream) (objectsDeleted, segmentsDeleted int64, err error)
	IterateZombieObjects(ctx context.Context, opts DeleteZombieObjects, process func(context.Context, []ObjectStream) error) (err error)
	DeleteInactiveObjectsAndSegments(ctx context.Context, objects []ObjectStream, opts DeleteZombieObjects) (objectsDeleted, segmentsDeleted int64, err error)
	DeleteAllBucketObjects(ctx context.Context, opts DeleteAllBucketObjects) (deletedObjectCount, deletedSegmentCount int64, err error)
	UncoordinatedDeleteAllBucketObjects(ctx context.Context, opts UncoordinatedDeleteAllBucketObjects) (deletedObjectCount, deletedSegmentCount int64, err error)

	EnsureNodeAliases(ctx context.Context, opts EnsureNodeAliases) error
	ListNodeAliases(ctx context.Context) (entries []NodeAliasEntry, err error)
	GetNodeAliasEntries(ctx context.Context, opts GetNodeAliasEntries) (entries []NodeAliasEntry, err error)
	GetStreamPieceCountByAlias(ctx context.Context, opts GetStreamPieceCountByNodeID) (result map[NodeAlias]int64, err error)

	TestingBatchInsertSegments(ctx context.Context, aliasCache *NodeAliasCache, segments []RawSegment) (err error)
	TestingGetAllObjects(ctx context.Context) (_ []RawObject, err error)
	TestingGetAllSegments(ctx context.Context, aliasCache *NodeAliasCache) (_ []RawSegment, err error)
	TestingDeleteAll(ctx context.Context) (err error)
	TestingBatchInsertObjects(ctx context.Context, objects []RawObject) (err error)
	TestingSetObjectVersion(ctx context.Context, object ObjectStream, randomVersion Version) (rowsAffected int64, err error)
	TestingSetPlacementAllSegments(ctx context.Context, placement storj.PlacementConstraint) (err error)
	TestingSetObjectCreatedAt(ctx context.Context, object ObjectStream, createdAt time.Time) (rowsAffected int64, err error)

	// TestMigrateToLatest creates a database and applies all the migration for test purposes.
	TestMigrateToLatest(ctx context.Context) error

	Config() *Config
	// contains filtered or unexported methods
}

Adapter is a low level extension point to use datasource related queries. TODO: we may need separated adapter for segments/objects/etc.

type AliasPiece 

type AliasPiece struct {
	Number uint16
	Alias  NodeAlias
}

AliasPiece is a piece with alias node ID.

type AliasPieces 

type AliasPieces []AliasPiece

AliasPieces is a slice of AliasPiece.

func (AliasPieces) Bytes 

func (aliases AliasPieces) Bytes() ([]byte, error)

Bytes compresses alias pieces to a slice of bytes.

func (*AliasPieces) DecodeSpanner added in v1.103.2 

func (aliases *AliasPieces) DecodeSpanner(val any) (err error)

DecodeSpanner implements spanner.Decoder.

func (AliasPieces) EncodeSpanner added in v1.103.2 

func (aliases AliasPieces) EncodeSpanner() (any, error)

EncodeSpanner implements spanner.Encoder.

func (*AliasPieces) Scan 

func (aliases *AliasPieces) Scan(src any) error

Scan implements the database/sql Scanner interface.

func (*AliasPieces) SetBytes 

func (aliases *AliasPieces) SetBytes(data []byte) error

SetBytes decompresses alias pieces from a slice of bytes.

func (AliasPieces) Value 

func (aliases AliasPieces) Value() (driver.Value, error)

Value implements the database/sql/driver Valuer interface.

type BeginCopyObject added in v1.50.1 

type BeginCopyObject struct {
	ObjectLocation
	Version Version

	SegmentLimit int64

	// VerifyLimits holds a callback by which the caller can interrupt the copy
	// if it turns out the copy would exceed a limit.
	VerifyLimits func(encryptedObjectSize int64, nSegments int64) error
}

BeginCopyObject holds all data needed begin copy object method.

type BeginCopyObjectResult added in v1.50.1 

type BeginCopyObjectResult BeginMoveCopyResults

BeginCopyObjectResult holds data needed to begin copy object.

type BeginMoveCopyResults added in v1.65.1 

type BeginMoveCopyResults struct {
	StreamID uuid.UUID
	Version  Version
	EncryptedUserData
	EncryptedKeysNonces  []EncryptedKeyAndNonce
	EncryptionParameters storj.EncryptionParameters
}

BeginMoveCopyResults holds all data needed to begin move and copy object methods.

type BeginMoveObject added in v1.39.4 

type BeginMoveObject struct {
	ObjectLocation

	SegmentLimit int64
}

BeginMoveObject holds all data needed begin move object method.

type BeginMoveObjectResult added in v1.39.4 

type BeginMoveObjectResult BeginMoveCopyResults

BeginMoveObjectResult holds data needed to begin move object.

type BeginObjectExactVersion 

type BeginObjectExactVersion struct {
	ObjectStream

	ExpiresAt              *time.Time
	ZombieDeletionDeadline *time.Time

	EncryptedUserData
	Encryption storj.EncryptionParameters

	Retention Retention // optional
	LegalHold bool

	// TestingBypassVerify makes the (*DB).TestingBeginObjectExactVersion method skip
	// validation of this struct's fields. This is useful for inserting intentionally
	// malformed or unexpected data into the database and testing that we handle it properly.
	TestingBypassVerify bool

	// supported only by Spanner.
	MaxCommitDelay *time.Duration
}

BeginObjectExactVersion contains arguments necessary for starting an object upload.

func (*BeginObjectExactVersion) Verify added in v1.43.1 

func (opts *BeginObjectExactVersion) Verify() error

Verify verifies get object reqest fields.

type BeginObjectNextVersion 

type BeginObjectNextVersion struct {
	ObjectStream

	ExpiresAt              *time.Time
	ZombieDeletionDeadline *time.Time

	EncryptedUserData
	Encryption storj.EncryptionParameters

	Retention Retention // optional
	LegalHold bool

	// supported only by Spanner.
	MaxCommitDelay *time.Duration
}

BeginObjectNextVersion contains arguments necessary for starting an object upload.

func (*BeginObjectNextVersion) Verify added in v1.43.1 

func (opts *BeginObjectNextVersion) Verify() error

Verify verifies get object request fields.

type BeginSegment 

type BeginSegment struct {
	ObjectStream

	Position SegmentPosition

	// TODO: unused field, can remove
	RootPieceID storj.PieceID

	Pieces Pieces

	ObjectExistsChecked bool
}

BeginSegment contains options to verify, whether a new segment upload can be started.

type BucketEmpty 

type BucketEmpty struct {
	ProjectID  uuid.UUID
	BucketName BucketName
}

BucketEmpty contains arguments necessary for checking if bucket is empty.

type BucketLocation 

type BucketLocation struct {
	ProjectID  uuid.UUID
	BucketName BucketName
}

BucketLocation defines a bucket that belongs to a project.

func ParseBucketPrefix 

func ParseBucketPrefix(prefix BucketPrefix) (BucketLocation, error)

ParseBucketPrefix parses BucketPrefix.

func ParseCompactBucketPrefix 

func ParseCompactBucketPrefix(compactPrefix []byte) (BucketLocation, error)

ParseCompactBucketPrefix parses BucketPrefix.

func (BucketLocation) CompactPrefix 

func (loc BucketLocation) CompactPrefix() []byte

CompactPrefix converts bucket location into bucket prefix with compact project ID.

func (BucketLocation) Compare added in v1.107.1 

func (loc BucketLocation) Compare(other BucketLocation) int

Compare compares this BucketLocation with another.

func (BucketLocation) Prefix 

func (loc BucketLocation) Prefix() BucketPrefix

Prefix converts bucket location into bucket prefix.

func (BucketLocation) Verify 

func (loc BucketLocation) Verify() error

Verify object location fields.

type BucketName added in v1.109.1 

type BucketName string

BucketName is a plain-text string, however we should treat it as unsafe bytes to avoid issues with any encoding.

func (BucketName) Compare added in v1.109.1 

func (b BucketName) Compare(x BucketName) int

Compare implements comparison for bucket names.

func (*BucketName) DecodeSpanner added in v1.109.1 

func (b *BucketName) DecodeSpanner(value any) error

DecodeSpanner implements spanner.Decoder.

func (BucketName) EncodeSpanner added in v1.109.1 

func (b BucketName) EncodeSpanner() (any, error)

EncodeSpanner implements spanner.Encoder.

func (*BucketName) Scan added in v1.109.1 

func (b *BucketName) Scan(value interface{}) error

Scan extracts a BucketName from a database field.

func (BucketName) String added in v1.109.1 

func (b BucketName) String() string

String implements stringer func.

func (BucketName) Value added in v1.109.1 

func (b BucketName) Value() (driver.Value, error)

Value converts a BucketName to a database field.

type BucketPrefix 

type BucketPrefix string

BucketPrefix consists of <project id>/<bucket name>.

type BucketTally added in v1.66.1 

type BucketTally struct {
	BucketLocation

	ObjectCount        int64
	PendingObjectCount int64

	TotalSegments int64
	TotalBytes    int64

	MetadataSize int64

	// BytesByRemainder maps storage remainder values to total bytes calculated with that remainder.
	// The map key is the remainder value in bytes, and the value is the total bytes for this bucket
	// calculated with that remainder applied.
	BytesByRemainder map[int64]int64
}

BucketTally contains information about aggregate data stored in a bucket.

type CockroachAdapter added in v1.102.2 

type CockroachAdapter struct {
	PostgresAdapter
}

CockroachAdapter uses Cockroach related SQL queries.

func (*CockroachAdapter) CommitInlineSegment added in v1.105.2 

func (p *CockroachAdapter) CommitInlineSegment(ctx context.Context, opts CommitInlineSegment) (err error)

CommitInlineSegment commits inline segment to the database.

func (*CockroachAdapter) CommitPendingObjectSegment added in v1.104.1 

func (p *CockroachAdapter) CommitPendingObjectSegment(ctx context.Context, opts CommitSegment, aliasPieces AliasPieces) (err error)

CommitPendingObjectSegment commits segment to the database.

func (*CockroachAdapter) DeleteAllBucketObjects added in v1.116.3 

func (c *CockroachAdapter) DeleteAllBucketObjects(ctx context.Context, opts DeleteAllBucketObjects) (totalDeletedObjects, totalDeletedSegments int64, err error)

DeleteAllBucketObjects deletes objects in the specified bucket in batches of opts.BatchSize number of objects.

func (*CockroachAdapter) GetTableStats added in v1.104.1 

func (c *CockroachAdapter) GetTableStats(ctx context.Context, opts GetTableStats) (result TableStats, err error)

GetTableStats implements Adapter.

func (*CockroachAdapter) IterateLoopSegments added in v1.114.5 

func (c *CockroachAdapter) IterateLoopSegments(ctx context.Context, aliasCache *NodeAliasCache, opts IterateLoopSegments, fn func(context.Context, LoopSegmentsIterator) error) (err error)

IterateLoopSegments implements Adapter.

func (*CockroachAdapter) MigrateToLatest added in v1.112.2 

func (c *CockroachAdapter) MigrateToLatest(ctx context.Context) error

MigrateToLatest migrates database to the latest version.

func (*CockroachAdapter) Name added in v1.106.1 

func (c *CockroachAdapter) Name() string

Name returns the name of the adapter.

func (*CockroachAdapter) TestMigrateToLatest added in v1.110.1 

func (c *CockroachAdapter) TestMigrateToLatest(ctx context.Context) error

TestMigrateToLatest creates a database and applies all the migration for test purposes.

func (*CockroachAdapter) UpdateTableStats added in v1.107.1 

func (c *CockroachAdapter) UpdateTableStats(ctx context.Context) error

UpdateTableStats forces an update of table statistics. Probably useful mostly in test scenarios.

type CollectBucketTallies added in v1.66.1 

type CollectBucketTallies struct {
	From               BucketLocation
	To                 BucketLocation
	AsOfSystemTime     time.Time
	AsOfSystemInterval time.Duration
	Now                time.Time

	UsePartitionQuery bool

	// StorageRemainders is a list of remainder values to calculate for each bucket.
	// Objects with total_encrypted_size less than a remainder value are counted as that remainder value.
	// Results are returned in BucketTally.BytesByRemainder map.
	//
	// Example: []int64{0, 51200, 102400} will calculate three values per bucket:
	//   - BytesByRemainder[0] = actual total size
	//   - BytesByRemainder[51200] = total size with 50KB minimum per object
	//   - BytesByRemainder[102400] = total size with 100KB minimum per object
	//
	// An empty list defaults to []int64{0} (no remainder applied).
	StorageRemainders []int64
}

CollectBucketTallies contains arguments necessary for looping through objects in metabase.

func (*CollectBucketTallies) Verify added in v1.66.1 

func (opts *CollectBucketTallies) Verify() error

Verify verifies CollectBucketTallies request fields.

type CommitInlineObject added in v1.106.1 

type CommitInlineObject struct {
	ObjectStream
	CommitInlineSegment CommitInlineSegment

	ExpiresAt *time.Time

	EncryptedUserData
	Encryption storj.EncryptionParameters

	Retention Retention // optional
	LegalHold bool

	DisallowDelete bool

	// Versioned indicates whether an object is allowed to have multiple versions.
	Versioned bool

	// IfNoneMatch is an optional field for conditional writes.
	IfNoneMatch IfNoneMatch

	// supported only by Spanner.
	TransmitEvent bool
}

CommitInlineObject contains arguments necessary for committing an inline object.

func (*CommitInlineObject) Verify added in v1.106.1 

func (c *CommitInlineObject) Verify() error

Verify verifies reqest fields.

type CommitInlineSegment 

type CommitInlineSegment struct {
	ObjectStream

	Position SegmentPosition

	ExpiresAt *time.Time

	EncryptedKeyNonce []byte
	EncryptedKey      []byte

	PlainOffset   int64 // offset in the original data stream
	PlainSize     int32 // size before encryption
	EncryptedETag []byte

	InlineData []byte

	SkipPendingObject bool

	// supported only by Spanner.
	MaxCommitDelay *time.Duration
}

CommitInlineSegment contains all necessary information about the segment.

func (CommitInlineSegment) Verify added in v1.106.1 

func (opts CommitInlineSegment) Verify() error

Verify verifies commit inline segment reqest fields.

type CommitObject 

type CommitObject struct {
	ObjectStream

	Encryption storj.EncryptionParameters
	ExpiresAt  *time.Time

	// OverrideEncryptedMedata flag controls if we want to set metadata fields with CommitObject
	// it's possible to set metadata with BeginObject request so we need to
	// be explicit if we would like to set it with CommitObject which will
	// override any existing metadata.
	OverrideEncryptedMetadata bool
	EncryptedUserData

	// Retention and LegalHold are used only for regular uploads (when SkipPendingObject is true).
	// For multipart uploads, these values are retrieved from the pending object in the database.
	Retention Retention // optional
	LegalHold bool

	// TODO: maybe this should use segment ranges rather than individual items
	SpecificSegments bool
	OnlySegments     []SegmentPosition

	DisallowDelete bool

	// Versioned indicates whether an object is allowed to have multiple versions.
	Versioned bool

	// supported only by Spanner.
	MaxCommitDelay *time.Duration
	TransmitEvent  bool

	// IfNoneMatch is an optional field for conditional writes.
	IfNoneMatch IfNoneMatch

	// SkipPendingObject indicates whether to skip checking for the existence of a pending object.
	// It's used for regular (non-multipart) uploads where we directly commit the object without a prior pending state.
	SkipPendingObject bool
}

CommitObject contains arguments necessary for committing an object.

func (*CommitObject) Verify added in v1.43.1 

func (c *CommitObject) Verify() error

Verify verifies request fields.

type CommitSegment 

type CommitSegment struct {
	ObjectStream

	Position    SegmentPosition
	RootPieceID storj.PieceID

	ExpiresAt *time.Time

	EncryptedKeyNonce []byte
	EncryptedKey      []byte

	PlainOffset   int64 // offset in the original data stream
	PlainSize     int32 // size before encryption
	EncryptedSize int32 // segment size after encryption

	EncryptedETag []byte

	Redundancy storj.RedundancyScheme

	Pieces Pieces

	Placement storj.PlacementConstraint

	// supported only by Spanner.
	MaxCommitDelay *time.Duration

	SkipPendingObject bool

	TestingUseMutations bool
}

CommitSegment contains all necessary information about the segment.

type Config added in v1.42.2 

type Config struct {
	ApplicationName  string
	MinPartSize      memory.Size
	MaxNumberOfParts int

	// TODO remove this flag when server-side copy implementation will be finished
	ServerSideCopy         bool
	ServerSideCopyDisabled bool

	TestingUniqueUnversioned bool
	TestingSpannerProjects   map[uuid.UUID]struct{}
	TestingWrapAdapter       func(Adapter) Adapter
	// TestingTimestampVersioning uses timestamps for assigning version numbers.
	TestingTimestampVersioning bool
	// TestingSpannerMinOpenedSessions allows to override the minimum number of sessions that client tries to keep open.
	TestingSpannerMinOpenedSessions *int

	Compression string

	FlightRecorder *flightrecorder.Box
}

Config is a configuration struct for part validation.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB implements a database for storing objects and segments.

func Open 

func Open(ctx context.Context, log *zap.Logger, connstr string, config Config) (*DB, error)

Open opens a connection to metabase.

func OpenDatabaseWithMigration added in v1.120.1 

func OpenDatabaseWithMigration(ctx context.Context, logger *zap.Logger, cfg DatabaseConfig) (*DB, error)

OpenDatabaseWithMigration will open the database (and update schema, if required).

func (*DB) BeginCopyObject added in v1.50.1 

func (db *DB) BeginCopyObject(ctx context.Context, opts BeginCopyObject) (_ BeginCopyObjectResult, err error)

BeginCopyObject collects all data needed to begin object copy procedure.

func (*DB) BeginMoveObject added in v1.39.4 

func (db *DB) BeginMoveObject(ctx context.Context, opts BeginMoveObject) (_ BeginMoveObjectResult, err error)

BeginMoveObject collects all data needed to begin object move procedure.

func (*DB) BeginObjectExactVersion 

func (db *DB) BeginObjectExactVersion(ctx context.Context, opts BeginObjectExactVersion) (committed Object, err error)

BeginObjectExactVersion adds a pending object to the database, with specific version.

func (*DB) BeginObjectNextVersion 

func (db *DB) BeginObjectNextVersion(ctx context.Context, opts BeginObjectNextVersion) (object Object, err error)

BeginObjectNextVersion adds a pending object to the database, with automatically assigned version.

func (*DB) BeginSegment 

func (db *DB) BeginSegment(ctx context.Context, opts BeginSegment) (err error)

BeginSegment verifies, whether a new segment upload can be started.

func (*DB) BucketEmpty 

func (db *DB) BucketEmpty(ctx context.Context, opts BucketEmpty) (empty bool, err error)

BucketEmpty returns true if bucket does not contain objects (pending or committed). This method doesn't check bucket existence.

func (*DB) CheckSegmentPiecesAlteration added in v1.132.5 

func (db *DB) CheckSegmentPiecesAlteration(
	ctx context.Context, streamID uuid.UUID, position SegmentPosition, pieces Pieces,
) (altered bool, err error)

CheckSegmentPiecesAlteration checks if the segment with streamID, and position is present in the DB and its pieces match pieces.

It returns true if the pieces don't match, otherwise false.

It returns an error of class `ErrSegmentNotFound` if the segment doesn't exist in the DB or any other type if there is another kind of error.

func (*DB) CheckVersion 

func (db *DB) CheckVersion(ctx context.Context) error

CheckVersion checks the database is the correct version.

func (*DB) ChooseAdapter added in v1.102.2 

func (db *DB) ChooseAdapter(projectID uuid.UUID) Adapter

ChooseAdapter selects the right adapter based on configuration.

func (*DB) Close 

func (db *DB) Close() error

Close closes the connection to database.

func (*DB) CollectBucketTallies added in v1.66.1 

func (db *DB) CollectBucketTallies(ctx context.Context, opts CollectBucketTallies) (result []BucketTally, err error)

CollectBucketTallies collect limited bucket tallies from given bucket locations.

func (*DB) CommitInlineObject added in v1.106.1 

func (db *DB) CommitInlineObject(ctx context.Context, opts CommitInlineObject) (object Object, err error)

CommitInlineObject adds full inline object to the database.

func (*DB) CommitInlineSegment 

func (db *DB) CommitInlineSegment(ctx context.Context, opts CommitInlineSegment) (err error)

CommitInlineSegment commits inline segment to the database.

func (*DB) CommitObject 

func (db *DB) CommitObject(ctx context.Context, opts CommitObject) (object Object, err error)

CommitObject adds a pending object to the database.

func (*DB) CommitSegment 

func (db *DB) CommitSegment(ctx context.Context, opts CommitSegment) (err error)

CommitSegment commits segment to the database.

func (*DB) CountSegments added in v1.123.4 

func (db *DB) CountSegments(ctx context.Context, checkTimestamp time.Time) (result SegmentsStats, err error)

CountSegments returns the number of segments in the segments table.

func (*DB) DeleteAllBucketObjects added in v1.116.3 

func (db *DB) DeleteAllBucketObjects(ctx context.Context, opts DeleteAllBucketObjects) (deletedObjects int64, err error)

DeleteAllBucketObjects deletes all objects in the specified bucket. Deletion performs in batches, so in case of error while processing, this method will return the number of objects deleted to the moment when an error occurs.

func (*DB) DeleteExpiredObjects 

func (db *DB) DeleteExpiredObjects(ctx context.Context, opts DeleteExpiredObjects) (err error)

DeleteExpiredObjects deletes all objects that expired before expiredBefore.

func (*DB) DeleteObjectExactVersion 

func (db *DB) DeleteObjectExactVersion(ctx context.Context, opts DeleteObjectExactVersion) (result DeleteObjectResult, err error)

DeleteObjectExactVersion deletes an exact object version.

func (*DB) DeleteObjectLastCommitted added in v1.63.1 

func (db *DB) DeleteObjectLastCommitted(
	ctx context.Context, opts DeleteObjectLastCommitted,
) (result DeleteObjectResult, err error)

DeleteObjectLastCommitted deletes an object last committed version.

func (*DB) DeleteObjectLastCommittedSuspended added in v1.142.7 

func (db *DB) DeleteObjectLastCommittedSuspended(ctx context.Context, opts DeleteObjectLastCommitted, deleterMarkerStreamID uuid.UUID) (result DeleteObjectResult, err error)

DeleteObjectLastCommittedSuspended deletes an object last committed version when opts.Suspended is true.

func (*DB) DeleteObjects added in v1.120.1 

func (db *DB) DeleteObjects(ctx context.Context, opts DeleteObjects) (result DeleteObjectsResult, err error)

DeleteObjects deletes specific objects from a bucket.

func (*DB) DeletePendingObject 

func (db *DB) DeletePendingObject(ctx context.Context, opts DeletePendingObject) (result DeleteObjectResult, err error)

DeletePendingObject deletes a pending object with specified version and streamID.

func (*DB) DeleteZombieObjects added in v1.30.1 

func (db *DB) DeleteZombieObjects(ctx context.Context, opts DeleteZombieObjects) (err error)

DeleteZombieObjects deletes all objects that zombie deletion deadline passed. TODO will be removed when objects table will be free from pending objects.

func (*DB) EnsureNodeAliases 

func (db *DB) EnsureNodeAliases(ctx context.Context, opts EnsureNodeAliases) (err error)

EnsureNodeAliases ensures that the supplied node ID-s have a alias. It's safe to concurrently try and create node ID-s for the same NodeID.

func (*DB) FinishCopyObject added in v1.50.1 

func (db *DB) FinishCopyObject(ctx context.Context, opts FinishCopyObject) (object Object, err error)

FinishCopyObject accepts new encryption keys for copied object and insert the corresponding new object ObjectKey and segments EncryptedKey. It returns the object at the destination location.

func (*DB) FinishMoveObject added in v1.40.3 

func (db *DB) FinishMoveObject(ctx context.Context, opts FinishMoveObject) (err error)

FinishMoveObject accepts new encryption keys for moved object and updates the corresponding object ObjectKey and segments EncryptedKey.

func (*DB) GetLatestObjectLastSegment 

func (db *DB) GetLatestObjectLastSegment(ctx context.Context, opts GetLatestObjectLastSegment) (segment Segment, err error)

GetLatestObjectLastSegment returns an object last segment information.

func (*DB) GetNodeAliasEntries added in v1.108.1 

func (db *DB) GetNodeAliasEntries(ctx context.Context, opts GetNodeAliasEntries) (entries []NodeAliasEntry, err error)

GetNodeAliasEntries fetches node aliases or ID-s for the specified nodes and aliases in random order.

func (*DB) GetObjectExactVersion 

func (db *DB) GetObjectExactVersion(ctx context.Context, opts GetObjectExactVersion) (_ Object, err error)

GetObjectExactVersion returns object information for exact version.

func (*DB) GetObjectExactVersionLegalHold added in v1.113.1 

func (db *DB) GetObjectExactVersionLegalHold(ctx context.Context, opts GetObjectExactVersionLegalHold) (enabled bool, err error)

GetObjectExactVersionLegalHold returns the legal hold configuration of an exact version of an object.

func (*DB) GetObjectExactVersionRetention added in v1.110.1 

func (db *DB) GetObjectExactVersionRetention(ctx context.Context, opts GetObjectExactVersionRetention) (retention Retention, err error)

GetObjectExactVersionRetention returns the retention configuration of an exact version of an object.

func (*DB) GetObjectLastCommitted added in v1.62.1 

func (db *DB) GetObjectLastCommitted(ctx context.Context, opts GetObjectLastCommitted) (_ Object, err error)

GetObjectLastCommitted returns object information for last committed version.

func (*DB) GetObjectLastCommittedLegalHold added in v1.113.1 

func (db *DB) GetObjectLastCommittedLegalHold(ctx context.Context, opts GetObjectLastCommittedLegalHold) (enabled bool, err error)

GetObjectLastCommittedLegalHold returns the legal hold configuration of the most recently committed version of an object.

func (*DB) GetObjectLastCommittedRetention added in v1.110.1 

func (db *DB) GetObjectLastCommittedRetention(ctx context.Context, opts GetObjectLastCommittedRetention) (retention Retention, err error)

GetObjectLastCommittedRetention returns the retention configuration of the most recently committed version of an object.

func (*DB) GetSegmentByPosition 

func (db *DB) GetSegmentByPosition(ctx context.Context, opts GetSegmentByPosition) (segment Segment, err error)

GetSegmentByPosition returns information about segment on the specified position.

func (*DB) GetSegmentByPositionForAudit added in v1.131.3 

func (db *DB) GetSegmentByPositionForAudit(
	ctx context.Context, opts GetSegmentByPosition,
) (segment SegmentForAudit, err error)

GetSegmentByPositionForAudit returns information about segment on the specified position for the audit functionality.

func (*DB) GetSegmentByPositionForRepair added in v1.131.3 

func (db *DB) GetSegmentByPositionForRepair(
	ctx context.Context, opts GetSegmentByPosition,
) (segment SegmentForRepair, err error)

GetSegmentByPositionForRepair returns information about segment on the specified position for the repair functionality.

func (*DB) GetStreamPieceCountByNodeID 

func (db *DB) GetStreamPieceCountByNodeID(ctx context.Context, opts GetStreamPieceCountByNodeID) (result map[storj.NodeID]int64, err error)

GetStreamPieceCountByNodeID returns piece count by node id.

func (*DB) GetTableStats added in v1.32.2 

func (db *DB) GetTableStats(ctx context.Context, opts GetTableStats) (result TableStats, err error)

GetTableStats gathers information about the metabase tables, currently only "segments" table.

func (*DB) Implementation added in v1.32.2 

func (db *DB) Implementation() dbutil.Implementation

Implementation returns the implementation for the first db adapter. TODO: remove this.

func (*DB) IterateLoopSegments added in v1.31.1 

func (db *DB) IterateLoopSegments(ctx context.Context, opts IterateLoopSegments, fn func(context.Context, LoopSegmentsIterator) error) (err error)

IterateLoopSegments iterates through all segments in metabase.

func (*DB) IterateObjectsAllVersionsWithStatus 

func (db *DB) IterateObjectsAllVersionsWithStatus(ctx context.Context, opts IterateObjectsWithStatus, fn func(context.Context, ObjectsIterator) error) (err error)

IterateObjectsAllVersionsWithStatus iterates through all versions of all objects with specified status.

func (*DB) IterateObjectsAllVersionsWithStatusAscending added in v1.96.4 

func (db *DB) IterateObjectsAllVersionsWithStatusAscending(ctx context.Context, opts IterateObjectsWithStatus, fn func(context.Context, ObjectsIterator) error) (err error)

IterateObjectsAllVersionsWithStatusAscending iterates through all versions of all objects with specified status. Ordered from oldest to latest. TODO this method was copied (and renamed) from v1.95.1 as a workaround for issues with metabase.ListObject performance. It should be removed when problem with metabase.ListObject will be fixed.

func (*DB) IteratePendingObjectsByKey 

func (db *DB) IteratePendingObjectsByKey(ctx context.Context, opts IteratePendingObjectsByKey, fn func(context.Context, ObjectsIterator) error) (err error)

IteratePendingObjectsByKey iterates through all streams of pending objects with the same ObjectKey.

func (*DB) LatestNodesAliasMap added in v1.66.1 

func (db *DB) LatestNodesAliasMap(ctx context.Context) (_ *NodeAliasMap, err error)

LatestNodesAliasMap returns the latest mapping between storj.NodeID and NodeAlias.

func (*DB) ListBucketStreamIDs added in v1.130.1 

func (db *DB) ListBucketStreamIDs(ctx context.Context, opts ListBucketStreamIDs, f func(ctx context.Context, streamIDs []uuid.UUID) error) (err error)

ListBucketStreamIDs lists the streamIDs from a bucket.

func (*DB) ListNodeAliases 

func (db *DB) ListNodeAliases(ctx context.Context) (_ []NodeAliasEntry, err error)

ListNodeAliases lists all node alias mappings.

func (*DB) ListObjects added in v1.65.1 

func (db *DB) ListObjects(ctx context.Context, opts ListObjects) (result ListObjectsResult, err error)

ListObjects lists objects.

func (*DB) ListSegments 

func (db *DB) ListSegments(ctx context.Context, opts ListSegments) (result ListSegmentsResult, err error)

ListSegments lists specified stream segments.

func (*DB) ListStreamPositions 

func (db *DB) ListStreamPositions(ctx context.Context, opts ListStreamPositions) (result ListStreamPositionsResult, err error)

ListStreamPositions lists specified stream segment positions.

func (*DB) ListVerifySegments added in v1.64.1 

func (db *DB) ListVerifySegments(ctx context.Context, opts ListVerifySegments) (result ListVerifySegmentsResult, err error)

ListVerifySegments lists specified stream segments.

func (*DB) MigrateToLatest 

func (db *DB) MigrateToLatest(ctx context.Context) error

MigrateToLatest migrates database to the latest version.

func (*DB) Now added in v1.30.1 

func (db *DB) Now(ctx context.Context) (time.Time, error)

Now returns the current time according to the first database adapter.

func (*DB) Ping 

func (db *DB) Ping(ctx context.Context) error

Ping checks whether connection has been established to all adapters.

func (*DB) PrecommitQuery added in v1.140.3 

func (db *DB) PrecommitQuery(ctx context.Context, opts PrecommitQuery, adapter precommitTransactionAdapter) (result *PrecommitInfo, err error)

PrecommitQuery queries all information about the object so it can be committed.

func (*DB) SetObjectExactVersionLegalHold added in v1.113.1 

func (db *DB) SetObjectExactVersionLegalHold(ctx context.Context, opts SetObjectExactVersionLegalHold) (err error)

SetObjectExactVersionLegalHold sets the legal hold configuration of an exact version of an object.

func (*DB) SetObjectExactVersionRetention added in v1.110.1 

func (db *DB) SetObjectExactVersionRetention(ctx context.Context, opts SetObjectExactVersionRetention) (err error)

SetObjectExactVersionRetention sets the retention configuration of an exact version of an object.

func (*DB) SetObjectLastCommittedLegalHold added in v1.113.1 

func (db *DB) SetObjectLastCommittedLegalHold(ctx context.Context, opts SetObjectLastCommittedLegalHold) (err error)

SetObjectLastCommittedLegalHold sets the legal hold configuration of the most recently committed version of an object.

func (*DB) SetObjectLastCommittedRetention added in v1.110.1 

func (db *DB) SetObjectLastCommittedRetention(ctx context.Context, opts SetObjectLastCommittedRetention) (err error)

SetObjectLastCommittedRetention sets the retention configuration of the most recently committed version of an object.

func (*DB) TestMigrateToLatest added in v1.57.1 

func (db *DB) TestMigrateToLatest(ctx context.Context) error

TestMigrateToLatest replaces the migration steps with only one step to create metabase db. It is applied to all db adapters.

func (*DB) TestingAllObjects 

func (db *DB) TestingAllObjects(ctx context.Context) (objects []Object, err error)

TestingAllObjects gets all objects. Use only for testing purposes.

func (*DB) TestingAllSegments 

func (db *DB) TestingAllSegments(ctx context.Context) (segments []Segment, err error)

TestingAllSegments gets all segments. Use only for testing purposes.

func (*DB) TestingBatchInsertObjects added in v1.98.1 

func (db *DB) TestingBatchInsertObjects(ctx context.Context, objects []RawObject) (err error)

TestingBatchInsertObjects batch inserts objects for testing. This implementation does no verification on the correctness of objects.

func (*DB) TestingBatchInsertSegments added in v1.102.2 

func (db *DB) TestingBatchInsertSegments(ctx context.Context, segments []RawSegment) (err error)

TestingBatchInsertSegments batch inserts segments for testing. This implementation does no verification on the correctness of segments.

func (*DB) TestingDeleteAll 

func (db *DB) TestingDeleteAll(ctx context.Context) (err error)

TestingDeleteAll deletes all objects and segments from the database.

func (*DB) TestingGetState 

func (db *DB) TestingGetState(ctx context.Context) (_ *RawState, err error)

TestingGetState returns the state of the database.

func (*DB) TestingSetCleanup added in v1.31.1 

func (db *DB) TestingSetCleanup(cleanup func() error)

TestingSetCleanup is used to set the callback for cleaning up test database.

func (*DB) TestingSetObjectCreatedAt added in v1.147.4 

func (db *DB) TestingSetObjectCreatedAt(ctx context.Context, object ObjectStream, createdAt time.Time) (rowsAffected int64, err error)

TestingSetObjectCreatedAt sets the created_at of the object to the given value in tests.

func (*DB) TestingSetObjectVersion added in v1.112.2 

func (db *DB) TestingSetObjectVersion(ctx context.Context, object ObjectStream, randomVersion Version) (rowsAffected int64, err error)

TestingSetObjectVersion sets the version of the object to the given value.

func (*DB) TestingSetPlacementAllSegments added in v1.112.2 

func (db *DB) TestingSetPlacementAllSegments(ctx context.Context, placement storj.PlacementConstraint) (err error)

TestingSetPlacementAllSegments sets the placement of all segments to the given value.

func (*DB) UncoordinatedDeleteAllBucketObjects added in v1.132.2 

func (db *DB) UncoordinatedDeleteAllBucketObjects(ctx context.Context, opts UncoordinatedDeleteAllBucketObjects) (deletedObjects int64, err error)

UncoordinatedDeleteAllBucketObjects deletes all objects in the specified bucket.

This deletion does not force the operations across the tables to be synchronized, speeding up the deletion. If there are any ongoing uploads/downloads/deletes it may create zombie segments.

Currently there's no special implementation for Postgres and Cockroach.

func (*DB) UpdateObjectLastCommittedMetadata added in v1.91.2 

func (db *DB) UpdateObjectLastCommittedMetadata(ctx context.Context, opts UpdateObjectLastCommittedMetadata) (err error)

UpdateObjectLastCommittedMetadata updates an object metadata.

func (*DB) UpdateSegmentPieces 

func (db *DB) UpdateSegmentPieces(ctx context.Context, opts UpdateSegmentPieces) (err error)

UpdateSegmentPieces updates pieces for specified segment. If provided old pieces won't match current database state update will fail.

func (*DB) UpdateTableStats added in v1.107.1 

func (db *DB) UpdateTableStats(ctx context.Context) (err error)

UpdateTableStats forces an update of table statistics. Probably useful mostly in test scenarios.

type DatabaseConfig added in v1.120.1 

type DatabaseConfig struct {
	MigrationUnsafe string `` /* 305-byte string literal not displayed */
	URL             string
	Config
}

DatabaseConfig is the minimum required configuration for metabase.

type DeleteAllBucketObjects added in v1.116.3 

type DeleteAllBucketObjects struct {
	Bucket    BucketLocation
	BatchSize int

	MaxStaleness   time.Duration
	MaxCommitDelay *time.Duration

	// supported only by Spanner.
	TransmitEvent bool

	// OnObjectsDeleted is called per batch with object info for deleted objects in that batch.
	// When nil, object info is not collected.
	OnObjectsDeleted func([]DeleteObjectsInfo)
}

DeleteAllBucketObjects contains arguments for deleting a whole bucket.

type DeleteExpiredObjects 

type DeleteExpiredObjects struct {
	ExpiredBefore      time.Time
	AsOfSystemInterval time.Duration
	BatchSize          int
	DeleteConcurrency  int
}

DeleteExpiredObjects contains all the information necessary to delete expired objects and segments.

type DeleteObjectExactVersion 

type DeleteObjectExactVersion struct {
	Version        Version
	StreamIDSuffix StreamIDSuffix
	ObjectLocation

	ObjectLock ObjectLockDeleteOptions

	// supported only by Spanner.
	TransmitEvent bool
}

DeleteObjectExactVersion contains arguments necessary for deleting an exact version of object.

func (*DeleteObjectExactVersion) Verify 

func (obj *DeleteObjectExactVersion) Verify() error

Verify delete object fields.

type DeleteObjectLastCommitted added in v1.63.1 

type DeleteObjectLastCommitted struct {
	ObjectLocation

	Versioned bool
	Suspended bool

	ObjectLock ObjectLockDeleteOptions

	// supported only by Spanner.
	TransmitEvent bool
}

DeleteObjectLastCommitted contains arguments necessary for deleting last committed version of object.

func (*DeleteObjectLastCommitted) Verify added in v1.63.1 

func (obj *DeleteObjectLastCommitted) Verify() error

Verify delete object last committed fields.

type DeleteObjectResult 

type DeleteObjectResult struct {
	// Removed contains the list of objects that were removed from the metabase.
	Removed []Object
	// Markers contains the delete markers that were added.
	Markers []Object
	// DeletedSegmentCount is the number of segments that were deleted.
	DeletedSegmentCount int
}

DeleteObjectResult result of deleting object.

type DeleteObjects added in v1.120.1 

type DeleteObjects struct {
	ProjectID  uuid.UUID
	BucketName BucketName
	Items      []DeleteObjectsItem

	Versioned bool
	Suspended bool

	ObjectLock ObjectLockDeleteOptions

	// supported only by Spanner.
	TransmitEvent bool
}

DeleteObjects contains options for deleting multiple committed objects from a bucket.

func (DeleteObjects) Verify added in v1.120.1 

func (opts DeleteObjects) Verify() error

Verify verifies bucket object deletion request fields.

type DeleteObjectsInfo added in v1.121.2 

type DeleteObjectsInfo struct {
	StreamVersionID    StreamVersionID
	Status             ObjectStatus
	CreatedAt          time.Time
	TotalEncryptedSize int64
}

DeleteObjectsInfo contains information about an object that was deleted or a delete marker that was inserted as a result of processing a DeleteObjects request item.

type DeleteObjectsItem added in v1.120.1 

type DeleteObjectsItem struct {
	ObjectKey       ObjectKey
	StreamVersionID StreamVersionID
}

DeleteObjectsItem describes the location of an object in a bucket to be deleted.

type DeleteObjectsResult added in v1.120.1 

type DeleteObjectsResult struct {
	Items               []DeleteObjectsResultItem
	DeletedSegmentCount int64
}

DeleteObjectsResult contains the results of an attempt to delete specific objects from a bucket.

type DeleteObjectsResultItem added in v1.120.1 

type DeleteObjectsResultItem struct {
	ObjectKey                ObjectKey
	RequestedStreamVersionID StreamVersionID

	Removed *DeleteObjectsInfo
	Marker  *DeleteObjectsInfo

	Status storj.DeleteObjectsStatus
}

DeleteObjectsResultItem contains the result of an attempt to delete a specific object from a bucket.

type DeletePendingObject 

type DeletePendingObject struct {
	ObjectStream

	// supported only by Spanner.
	MaxCommitDelay *time.Duration
}

DeletePendingObject contains arguments necessary for deleting a pending object.

func (*DeletePendingObject) Verify 

func (opts *DeletePendingObject) Verify() error

Verify verifies delete pending object fields validity.

type DeleteZombieObjects added in v1.30.1 

type DeleteZombieObjects struct {
	DeadlineBefore     time.Time
	InactiveDeadline   time.Time
	AsOfSystemInterval time.Duration
	BatchSize          int
}

DeleteZombieObjects contains all the information necessary to delete zombie objects and segments.

type EncryptedKeyAndNonce added in v1.39.4 

type EncryptedKeyAndNonce struct {
	Position          SegmentPosition
	EncryptedKeyNonce []byte
	EncryptedKey      []byte
}

EncryptedKeyAndNonce holds single segment position, encrypted key and nonce.

type EncryptedUserData added in v1.132.2 

type EncryptedUserData struct {
	EncryptedMetadata             []byte
	EncryptedMetadataNonce        []byte
	EncryptedMetadataEncryptedKey []byte
	EncryptedETag                 []byte
}

EncryptedUserData contains user data that has been encrypted with the nonce and key.

func (EncryptedUserData) Verify added in v1.132.2 

func (opts EncryptedUserData) Verify() error

Verify checks whether the fields have been set correctly.

type EnsureNodeAliases 

type EnsureNodeAliases struct {
	Nodes []storj.NodeID
}

EnsureNodeAliases contains arguments necessary for creating NodeAlias-es.

type ExcludeFromPending added in v1.141.2 

type ExcludeFromPending struct {
	// Object indicates whether the entire object should be excluded from read.
	// We want to exclude it during segment commit where pending object was not
	// created at the beginning of upload.
	// Segments are not excluded in this case.
	Object bool
	// ExpiresAt indicates whether the expires_at field should be excluded from read
	// We want to exclude it during object commit where we know expiration value but
	// don't want to exclude it for copy/move operations.
	ExpiresAt bool
	// EncryptedUserData indicates whether encrypted user data fields should be excluded from read.
	// We want to exclude it during object commit when data is provided explicitly but
	// don't want to exclude it for copy/move operations.
	EncryptedUserData bool
}

ExcludeFromPending contains fields to exclude from the pending object.

type FinishCopyObject added in v1.50.1 

type FinishCopyObject struct {
	ObjectStream
	NewBucket             BucketName
	NewEncryptedObjectKey ObjectKey
	NewStreamID           uuid.UUID

	// OverrideMetadata specifies that EncryptedETag and EncryptedMetadata should be changed on the copied object.
	// Otherwise, only EncryptedMetadataNonce and EncryptedMetadataEncryptedKey are changed.
	OverrideMetadata     bool
	NewEncryptedUserData EncryptedUserData

	NewSegmentKeys []EncryptedKeyAndNonce

	// NewDisallowDelete indicates whether the user is allowed to delete an existing unversioned object.
	NewDisallowDelete bool

	// NewVersioned indicates that the object allows multiple versions.
	NewVersioned bool

	// Retention indicates retention settings of the object copy.
	Retention Retention
	// LegalHold indicates whether the object copy is under legal hold.
	LegalHold bool

	// VerifyLimits holds a callback by which the caller can interrupt the copy
	// if it turns out completing the copy would exceed a limit.
	// It will be called only once.
	VerifyLimits func(encryptedObjectSize int64, nSegments int64) error

	// IfNoneMatch is an optional field for conditional writes.
	IfNoneMatch IfNoneMatch

	// supported only by Spanner.
	TransmitEvent bool
}

FinishCopyObject holds all data needed to finish object copy.

func (FinishCopyObject) NewLocation added in v1.92.1 

func (finishCopy FinishCopyObject) NewLocation() ObjectLocation

NewLocation returns the new object location.

func (FinishCopyObject) Verify added in v1.50.1 

func (finishCopy FinishCopyObject) Verify() error

Verify verifies metabase.FinishCopyObject data.

type FinishMoveObject added in v1.40.3 

type FinishMoveObject struct {
	ObjectStream

	NewBucket             BucketName
	NewSegmentKeys        []EncryptedKeyAndNonce
	NewEncryptedObjectKey ObjectKey
	// Optional. Required if object has metadata.
	NewEncryptedMetadataNonce        storj.Nonce
	NewEncryptedMetadataEncryptedKey []byte

	// NewDisallowDelete indicates whether the user is allowed to delete an existing unversioned object.
	NewDisallowDelete bool

	// NewVersioned indicates that the object allows multiple versions.
	NewVersioned bool

	// Retention indicates retention settings of the moved object
	// version.
	Retention Retention
	// LegalHold indicates legal hold settings of the moved object
	// version.
	LegalHold bool

	// supported only by Spanner.
	TransmitEvent bool
}

FinishMoveObject holds all data needed to finish object move.

func (FinishMoveObject) NewLocation added in v1.92.1 

func (finishMove FinishMoveObject) NewLocation() ObjectLocation

NewLocation returns the new object location.

func (FinishMoveObject) Verify added in v1.40.3 

func (finishMove FinishMoveObject) Verify() error

Verify verifies metabase.FinishMoveObject data.

type GetLatestObjectLastSegment 

type GetLatestObjectLastSegment struct {
	ObjectLocation
}

GetLatestObjectLastSegment contains arguments necessary for fetching a last segment information.

type GetNodeAliasEntries added in v1.108.1 

type GetNodeAliasEntries struct {
	Nodes   []storj.NodeID
	Aliases []NodeAlias
}

GetNodeAliasEntries contains arguments necessary for fetching node alias entries.

type GetObjectExactVersion 

type GetObjectExactVersion struct {
	Version Version
	ObjectLocation
}

GetObjectExactVersion contains arguments necessary for fetching an information about exact object version.

func (*GetObjectExactVersion) Verify 

func (obj *GetObjectExactVersion) Verify() error

Verify verifies get object request fields.

type GetObjectExactVersionLegalHold added in v1.113.1 

type GetObjectExactVersionLegalHold struct {
	ObjectLocation
	Version Version
}

GetObjectExactVersionLegalHold contains arguments necessary for retrieving the legal hold configuration of an exact version of an object.

type GetObjectExactVersionRetention added in v1.110.1 

type GetObjectExactVersionRetention struct {
	ObjectLocation
	Version Version
}

GetObjectExactVersionRetention contains arguments necessary for retrieving the retention configuration of an exact version of an object.

type GetObjectLastCommitted added in v1.62.1 

type GetObjectLastCommitted struct {
	ObjectLocation
}

GetObjectLastCommitted contains arguments necessary for fetching an object information for last committed version.

type GetObjectLastCommittedLegalHold added in v1.113.1 

type GetObjectLastCommittedLegalHold struct {
	ObjectLocation
}

GetObjectLastCommittedLegalHold contains arguments necessary for retrieving the legal hold configuration of the most recently committed version of an object.

type GetObjectLastCommittedRetention added in v1.110.1 

type GetObjectLastCommittedRetention struct {
	ObjectLocation
}

GetObjectLastCommittedRetention contains arguments necessary for retrieving the retention configuration of the most recently committed version of an object.

type GetSegmentByPosition 

type GetSegmentByPosition struct {
	StreamID uuid.UUID
	Position SegmentPosition
}

GetSegmentByPosition contains arguments necessary for fetching a segment on specific position.

func (*GetSegmentByPosition) Verify 

func (seg *GetSegmentByPosition) Verify() error

Verify verifies get segment request fields.

type GetStreamPieceCountByNodeID 

type GetStreamPieceCountByNodeID struct {
	ProjectID uuid.UUID
	StreamID  uuid.UUID
}

GetStreamPieceCountByNodeID contains arguments for GetStreamPieceCountByNodeID.

type GetTableStats added in v1.32.2 

type GetTableStats struct {
	AsOfSystemInterval time.Duration
}

GetTableStats contains arguments necessary for getting table statistics.

type IfNoneMatch added in v1.128.3 

type IfNoneMatch []string

IfNoneMatch is an option for conditional writes.

Currently it only supports a single value of "*" (all), which means the commit should only succeed if the object doesn't exist, or is a delete marker.

In future we may support other values like ETag.

func (IfNoneMatch) All added in v1.128.3 

func (i IfNoneMatch) All() bool

All returns whether IfNoneMatch value is "*".

func (IfNoneMatch) Verify added in v1.128.3 

func (i IfNoneMatch) Verify() error

Verify verifies IfNoneMatch is correct.

S3 returns unimplemented errors if requesting any value other than "*" or empty for uploads, so we do the same here.

type IterateCursor 

type IterateCursor struct {
	Key     ObjectKey
	Version Version
}

IterateCursor is a cursor used during iteration through objects.

The cursor is exclusive.

type IterateLoopSegments added in v1.31.1 

type IterateLoopSegments struct {
	BatchSize            int
	StartStreamID        uuid.UUID
	EndStreamID          uuid.UUID
	AsOfSystemInterval   time.Duration
	SpannerReadTimestamp time.Time
	SpannerQueryType     string
}

IterateLoopSegments contains arguments necessary for listing segments in metabase.

func (*IterateLoopSegments) Verify added in v1.31.1 

func (opts *IterateLoopSegments) Verify() error

Verify verifies segments request fields.

type IterateObjectsWithStatus 

type IterateObjectsWithStatus struct {
	ProjectID  uuid.UUID
	BucketName BucketName
	Recursive  bool
	BatchSize  int
	Prefix     ObjectKey
	Delimiter  ObjectKey
	Cursor     IterateCursor
	Pending    bool

	IncludeCustomMetadata       bool
	IncludeSystemMetadata       bool
	IncludeETag                 bool
	IncludeETagOrCustomMetadata bool
}

IterateObjectsWithStatus contains arguments necessary for listing objects in a bucket.

func (*IterateObjectsWithStatus) Verify 

func (opts *IterateObjectsWithStatus) Verify() error

Verify verifies get object request fields.

type IteratePendingObjectsByKey 

type IteratePendingObjectsByKey struct {
	ObjectLocation
	BatchSize int
	Cursor    StreamIDCursor
}

IteratePendingObjectsByKey contains arguments necessary for listing pending objects by ObjectKey.

func (*IteratePendingObjectsByKey) Verify 

func (opts *IteratePendingObjectsByKey) Verify() error

Verify verifies get object request fields.

type ListBucketStreamIDs added in v1.130.1 

type ListBucketStreamIDs struct {
	Bucket BucketLocation
	Limit  int

	AsOfSystemInterval time.Duration
}

ListBucketStreamIDs contains arguments necessary for listing stream segments from bucket.

type ListObjects added in v1.65.1 

type ListObjects struct {
	ProjectID   uuid.UUID
	BucketName  BucketName
	Recursive   bool
	Limit       int
	Prefix      ObjectKey
	Delimiter   ObjectKey
	Cursor      ListObjectsCursor
	Pending     bool
	AllVersions bool

	IncludeCustomMetadata       bool
	IncludeSystemMetadata       bool
	IncludeETag                 bool
	IncludeETagOrCustomMetadata bool

	Unversioned bool
	Params      ListObjectsParams
}

ListObjects contains arguments necessary for listing objects.

For Pending = false, the versions are in descending order. For Pending = true, the versions are in ascending order.

If Delimiter is empty, it will default to "/".

func (*ListObjects) FirstVersion added in v1.106.1 

func (opts *ListObjects) FirstVersion() Version

FirstVersion returns the first object version we need to iterate given the list objects logic.

func (*ListObjects) StartCursor added in v1.106.1 

func (opts *ListObjects) StartCursor() (cursor ListObjectsCursor, ok bool)

StartCursor returns the starting object cursor for this listing. If no delimiter is specified, the delimiter is treated as if it is "/". If no objects can be listed with these options, it returns an empty cursor and false.

func (*ListObjects) Verify added in v1.65.1 

func (opts *ListObjects) Verify() error

Verify verifies get object request fields.

func (*ListObjects) VersionAscending added in v1.101.1 

func (opts *ListObjects) VersionAscending() bool

VersionAscending returns whether the versions in the result are in ascending order.

type ListObjectsCursor added in v1.65.1 

type ListObjectsCursor IterateCursor

ListObjectsCursor is a cursor used during iteration through objects.

type ListObjectsParams added in v1.119.12 

type ListObjectsParams struct {
	// VersionSkipRequery is a limit on how many versions to skip before requerying.
	VersionSkipRequery int
	// PrefixSkipRequery is a limit on how many same prefix to skip before requerying.
	PrefixSkipRequery int
	// QueryExtraForNonRecursive is how many extra entries to query for non-recursive.
	QueryExtraForNonRecursive int
	// MinBatchSize is the number of items to query at the same time.
	MinBatchSize int
}

ListObjectsParams contains flags for tuning the ListObjects query.

type ListObjectsResult added in v1.65.1 

type ListObjectsResult struct {
	Objects []ObjectEntry
	More    bool
}

ListObjectsResult result of listing objects.

type ListSegments 

type ListSegments struct {
	ProjectID uuid.UUID
	StreamID  uuid.UUID
	Cursor    SegmentPosition
	Limit     int

	Range *StreamRange
}

ListSegments contains arguments necessary for listing stream segments.

type ListSegmentsResult 

type ListSegmentsResult struct {
	Segments []Segment
	More     bool
}

ListSegmentsResult result of listing segments.

type ListStreamPositions 

type ListStreamPositions struct {
	ProjectID uuid.UUID
	StreamID  uuid.UUID
	Cursor    SegmentPosition
	Limit     int

	Range *StreamRange
}

ListStreamPositions contains arguments necessary for listing stream segments.

type ListStreamPositionsResult 

type ListStreamPositionsResult struct {
	Segments []SegmentPositionInfo
	More     bool
}

ListStreamPositionsResult result of listing segments.

type ListVerifyBucketList added in v1.70.1 

type ListVerifyBucketList struct {
	Buckets []BucketLocation
}

ListVerifyBucketList represents a list of buckets.

func (*ListVerifyBucketList) Add added in v1.70.1 

func (list *ListVerifyBucketList) Add(projectID uuid.UUID, bucketName BucketName)

Add adds a (projectID, bucketName) to the list of buckets to be checked.

type ListVerifySegments added in v1.64.1 

type ListVerifySegments struct {
	CursorStreamID uuid.UUID
	CursorPosition SegmentPosition

	StreamIDs []uuid.UUID
	Limit     int

	CreatedAfter  *time.Time
	CreatedBefore *time.Time

	AsOfSystemTime     time.Time
	AsOfSystemInterval time.Duration
}

ListVerifySegments contains arguments necessary for listing stream segments.

type ListVerifySegmentsResult added in v1.64.1 

type ListVerifySegmentsResult struct {
	Segments []VerifySegment
}

ListVerifySegmentsResult is the result of ListVerifySegments.

type LoopSegmentEntry 

type LoopSegmentEntry struct {
	StreamID      uuid.UUID
	Position      SegmentPosition
	CreatedAt     time.Time // non-nillable
	ExpiresAt     *time.Time
	RepairedAt    *time.Time // repair
	RootPieceID   storj.PieceID
	EncryptedSize int32 // size of the whole segment (not a piece)
	PlainOffset   int64 // verify
	PlainSize     int32 // verify
	AliasPieces   AliasPieces
	Redundancy    storj.RedundancyScheme
	Pieces        Pieces
	Placement     storj.PlacementConstraint
	Source        string
}

LoopSegmentEntry contains information about segment metadata needed by metainfo loop.

func (LoopSegmentEntry) Inline 

func (s LoopSegmentEntry) Inline() bool

Inline returns true if segment is inline.

type LoopSegmentsIterator added in v1.31.1 

type LoopSegmentsIterator interface {
	Next(ctx context.Context, item *LoopSegmentEntry) bool
}

LoopSegmentsIterator iterates over a sequence of LoopSegmentEntry items.

type NodeAlias 

type NodeAlias int32

NodeAlias is a metabase local alias for NodeID-s to reduce segment table size.

type NodeAliasCache 

type NodeAliasCache struct {
	// contains filtered or unexported fields
}

NodeAliasCache is a write-through cache for looking up node ID and alias mapping.

func NewNodeAliasCache 

func NewNodeAliasCache(db NodeAliasDB, fullRefresh bool) *NodeAliasCache

NewNodeAliasCache creates a new cache using the specified database.

func (*NodeAliasCache) Aliases 

func (cache *NodeAliasCache) Aliases(ctx context.Context, nodes []storj.NodeID) ([]NodeAlias, error)

Aliases returns node aliases corresponding to the node ID-s and returns an error when node is missing.

func (*NodeAliasCache) ConvertAliasesToPieces 

func (cache *NodeAliasCache) ConvertAliasesToPieces(ctx context.Context, aliasPieces AliasPieces) (_ Pieces, err error)

ConvertAliasesToPieces converts alias pieces to pieces.

func (*NodeAliasCache) EnsureAliases added in v1.65.1 

func (cache *NodeAliasCache) EnsureAliases(ctx context.Context, nodes []storj.NodeID) ([]NodeAlias, error)

EnsureAliases returns node aliases corresponding to the node ID-s, adding missing node ID-s to the database when needed.

func (*NodeAliasCache) EnsurePiecesToAliases added in v1.65.1 

func (cache *NodeAliasCache) EnsurePiecesToAliases(ctx context.Context, pieces Pieces) (_ AliasPieces, err error)

EnsurePiecesToAliases converts pieces to alias pieces and automatically adds storage node to alias table when necessary.

func (*NodeAliasCache) Latest added in v1.66.1 

func (cache *NodeAliasCache) Latest(ctx context.Context) (_ *NodeAliasMap, err error)

Latest returns the latest NodeAliasMap.

func (*NodeAliasCache) Nodes 

func (cache *NodeAliasCache) Nodes(ctx context.Context, aliases []NodeAlias) ([]storj.NodeID, error)

Nodes returns node ID-s corresponding to the aliases, refreshing the cache once when an alias is missing. This results in an error when the alias is not in the database.

type NodeAliasDB 

type NodeAliasDB interface {
	EnsureNodeAliases(ctx context.Context, opts EnsureNodeAliases) error
	ListNodeAliases(ctx context.Context) (_ []NodeAliasEntry, err error)
	GetNodeAliasEntries(ctx context.Context, opts GetNodeAliasEntries) (_ []NodeAliasEntry, err error)
}

NodeAliasDB is an interface for looking up node alises.

type NodeAliasEntry 

type NodeAliasEntry struct {
	ID    storj.NodeID
	Alias NodeAlias
}

NodeAliasEntry is a mapping between NodeID and NodeAlias.

type NodeAliasMap 

type NodeAliasMap struct {
	// contains filtered or unexported fields
}

NodeAliasMap contains bidirectional mapping between node ID and a NodeAlias.

The node ID to NodeAlias lookup is implemented as a map of 4-byte node ID prefixes to a linked list of node ID/alias pairs, so that the whole ID does not need to be hashed with each lookup.

func NewNodeAliasMap 

func NewNodeAliasMap(entries []NodeAliasEntry) *NodeAliasMap

NewNodeAliasMap creates a new alias map from the given entries.

func (*NodeAliasMap) Alias added in v1.66.1 

func (m *NodeAliasMap) Alias(node storj.NodeID) (x NodeAlias, ok bool)

Alias returns alias for the given node ID.

func (*NodeAliasMap) Aliases 

func (m *NodeAliasMap) Aliases(nodes []storj.NodeID) (xs []NodeAlias, missing []storj.NodeID)

Aliases returns aliases-s for the given node ID-s and node ID-s that are not in this map.

func (*NodeAliasMap) ContainsAll 

func (m *NodeAliasMap) ContainsAll(nodeIDs []storj.NodeID, nodeAliases []NodeAlias) bool

ContainsAll returns true when the table contains all entries.

func (*NodeAliasMap) Max added in v1.70.1 

func (m *NodeAliasMap) Max() NodeAlias

Max returns the largest node alias in this map, -1 otherwise. Contrast with Size.

func (*NodeAliasMap) Merge 

func (m *NodeAliasMap) Merge(other *NodeAliasMap)

Merge merges the other map into m.

func (*NodeAliasMap) Node 

func (m *NodeAliasMap) Node(alias NodeAlias) (x storj.NodeID, ok bool)

Node returns NodeID for the given alias.

func (*NodeAliasMap) Nodes 

func (m *NodeAliasMap) Nodes(aliases []NodeAlias) (xs []storj.NodeID, missing []NodeAlias)

Nodes returns NodeID-s for the given aliases and aliases that are not in this map.

func (*NodeAliasMap) Size 

func (m *NodeAliasMap) Size() int

Size returns the number of entries in this map. Contrast with Max.

type NullableObjectStatus added in v1.128.3 

type NullableObjectStatus struct {
	ObjectStatus ObjectStatus
	Valid        bool
}

NullableObjectStatus represents a nullable ObjectStatus type. TODO: replace with sql.Null[ObjectStatus] when we can use Go 1.22+

func (*NullableObjectStatus) Scan added in v1.128.3 

func (v *NullableObjectStatus) Scan(val any) error

Scan implements sql.Scanner interface.

type NullableVersion added in v1.128.3 

type NullableVersion struct {
	Version Version
	Valid   bool
}

NullableVersion represents a nullable Version type. TODO: replace with sql.Null[Version] when we can use Go 1.22+

func (*NullableVersion) Scan added in v1.128.3 

func (v *NullableVersion) Scan(val any) error

Scan implements sql.Scanner interface.

type Object 

type Object RawObject

Object object metadata. TODO define separated struct.

func (*Object) IsMigrated 

func (obj *Object) IsMigrated() bool

IsMigrated returns whether the object comes from PointerDB. Pointer objects are special that they are missing some information.

  • TotalPlainSize = 0 and FixedSegmentSize = 0.
  • Segment.PlainOffset = 0, Segment.PlainSize = 0

func (*Object) StreamVersionID added in v1.95.1 

func (obj *Object) StreamVersionID() StreamVersionID

StreamVersionID returns byte representation of object stream version id.

type ObjectEntry 

type ObjectEntry struct {
	IsPrefix bool
	IsLatest bool

	ObjectKey ObjectKey
	Version   Version
	StreamID  uuid.UUID

	CreatedAt time.Time
	ExpiresAt *time.Time

	Status       ObjectStatus
	SegmentCount int32

	EncryptedUserData

	TotalPlainSize     int64
	TotalEncryptedSize int64
	FixedSegmentSize   int32

	Encryption storj.EncryptionParameters
}

ObjectEntry contains information about an item in a bucket.

func (ObjectEntry) Less added in v1.96.2 

func (entry ObjectEntry) Less(other ObjectEntry) bool

Less implements sorting on object entries.

func (ObjectEntry) LessVersionAsc added in v1.100.2 

func (entry ObjectEntry) LessVersionAsc(other ObjectEntry) bool

LessVersionAsc implements sorting on object entries.

func (ObjectEntry) StreamVersionID added in v1.95.1 

func (entry ObjectEntry) StreamVersionID() StreamVersionID

StreamVersionID returns byte representation of object stream version id.

type ObjectKey 

type ObjectKey string

ObjectKey is an encrypted object key encoded using Path Component Encoding. It is not ascii safe.

func SkipPrefix added in v1.133.2 

func SkipPrefix(prefix ObjectKey) (next ObjectKey, ok bool)

SkipPrefix returns the lexicographically smallest object key that is greater than any key with the given prefix. If no such prefix exists, it returns "", false.

func (*ObjectKey) DecodeSpanner added in v1.104.1 

func (o *ObjectKey) DecodeSpanner(value any) error

DecodeSpanner implements spanner.Decoder.

func (ObjectKey) EncodeSpanner added in v1.104.1 

func (o ObjectKey) EncodeSpanner() (any, error)

EncodeSpanner implements spanner.Encoder.

func (*ObjectKey) Scan added in v1.41.1 

func (o *ObjectKey) Scan(value interface{}) error

Scan extracts a ObjectKey from a database field.

func (ObjectKey) Value added in v1.41.1 

func (o ObjectKey) Value() (driver.Value, error)

Value converts a ObjectKey to a database field.

type ObjectLocation 

type ObjectLocation struct {
	ProjectID  uuid.UUID
	BucketName BucketName
	ObjectKey  ObjectKey
}

ObjectLocation is decoded object key information.

func (ObjectLocation) Bucket 

func (obj ObjectLocation) Bucket() BucketLocation

Bucket returns bucket location this object belongs to.

func (ObjectLocation) Verify 

func (obj ObjectLocation) Verify() error

Verify object location fields.

type ObjectLockDeleteOptions added in v1.113.1 

type ObjectLockDeleteOptions struct {
	// Enabled indicates that locked objects should be protected from deletion.
	Enabled bool

	// BypassGovernance allows governance mode retention restrictions to be bypassed.
	BypassGovernance bool
}

ObjectLockDeleteOptions contains options specifying how objects that may be subject to Object Lock restrictions should be deleted.

type ObjectStatus 

type ObjectStatus byte

ObjectStatus defines the status that the object is in.

There are two types of objects:

  • Regular (i.e. Committed), which is used for storing data.
  • Delete Marker, which is used to show that an object has been deleted, while preserving older versions.

Each object can be in two states:

  • Pending, meaning that it's still being uploaded.
  • Committed, meaning it has finished uploading. Delete Markers are always considered committed, because they do not require committing.

There are two options for versioning:

  • Unversioned, there's only one allowed per project, bucket and encryption key.
  • Versioned, there can be any number of such objects for a given project, bucket and encryption key.

These lead to a few meaningful distinct statuses, listed below.

func (*ObjectStatus) DecodeSpanner added in v1.104.1 

func (status *ObjectStatus) DecodeSpanner(val any) (err error)

DecodeSpanner implements spanner.Decoder.

func (ObjectStatus) EncodeSpanner added in v1.104.1 

func (status ObjectStatus) EncodeSpanner() (any, error)

EncodeSpanner implements spanner.Encoder.

func (ObjectStatus) IsCommitted added in v1.110.1 

func (status ObjectStatus) IsCommitted() bool

IsCommitted returns whether the status indicates that an object is committed.

func (ObjectStatus) IsDeleteMarker added in v1.91.2 

func (status ObjectStatus) IsDeleteMarker() bool

IsDeleteMarker return whether the status is a delete marker.

func (ObjectStatus) IsPending added in v1.130.5 

func (status ObjectStatus) IsPending() bool

IsPending returns whether the status is pending.

func (ObjectStatus) IsUnversioned added in v1.110.1 

func (status ObjectStatus) IsUnversioned() bool

IsUnversioned returns whether the status indicates that an object is unversioned.

func (*ObjectStatus) Scan added in v1.128.3 

func (status *ObjectStatus) Scan(val any) error

Scan implements sql.Scanner interface.

func (ObjectStatus) String added in v1.96.2 

func (status ObjectStatus) String() string

String returns textual representation of status.

type ObjectStream 

type ObjectStream struct {
	ProjectID  uuid.UUID
	BucketName BucketName
	ObjectKey  ObjectKey
	Version    Version
	StreamID   uuid.UUID
}

ObjectStream uniquely defines an object and stream.

func (ObjectStream) Less added in v1.91.2 

func (obj ObjectStream) Less(b ObjectStream) bool

Less implements sorting on object streams. Where ProjectID asc, BucketName asc, ObjectKey asc, Version desc.

func (ObjectStream) LessVersionAsc added in v1.100.2 

func (obj ObjectStream) LessVersionAsc(b ObjectStream) bool

LessVersionAsc implements sorting on object streams. Where ProjectID asc, BucketName asc, ObjectKey asc, Version asc.

func (*ObjectStream) Location 

func (obj *ObjectStream) Location() ObjectLocation

Location returns object location.

func (*ObjectStream) StreamIDSuffix added in v1.123.4 

func (obj *ObjectStream) StreamIDSuffix() StreamIDSuffix

StreamIDSuffix returns the object's stream ID suffix.

func (*ObjectStream) Verify 

func (obj *ObjectStream) Verify() error

Verify object stream fields.

type ObjectsIterator 

type ObjectsIterator interface {
	Next(ctx context.Context, item *ObjectEntry) bool
}

ObjectsIterator iterates over a sequence of ObjectEntry items.

type ObjectsIteratorCursor added in v1.106.1 

type ObjectsIteratorCursor struct {
	Key       ObjectKey
	Version   Version
	StreamID  uuid.UUID
	Inclusive bool
}

ObjectsIteratorCursor is the current location in an objects iterator.

func FirstIterateCursor added in v1.106.1 

func FirstIterateCursor(recursive bool, cursor IterateCursor, prefix, delimiter ObjectKey) (_ ObjectsIteratorCursor, ok bool)

FirstIterateCursor adjust the cursor for a non-recursive iteration. The cursor is non-inclusive and we need to adjust to handle prefix as cursor properly. We return the next possible key from the prefix.

type PendingObjectEntry added in v1.87.1 

type PendingObjectEntry struct {
	IsPrefix bool

	ObjectKey ObjectKey
	StreamID  uuid.UUID

	CreatedAt time.Time
	ExpiresAt *time.Time

	EncryptedMetadataNonce        []byte
	EncryptedMetadata             []byte
	EncryptedMetadataEncryptedKey []byte
	EncryptedETag                 []byte

	Encryption storj.EncryptionParameters
}

PendingObjectEntry contains information about an pending object item in a bucket.

type PendingObjectStream added in v1.85.1 

type PendingObjectStream struct {
	ProjectID  uuid.UUID
	BucketName BucketName
	ObjectKey  ObjectKey
	StreamID   uuid.UUID
}

PendingObjectStream uniquely defines an pending object and stream.

func (*PendingObjectStream) Verify added in v1.85.1 

func (obj *PendingObjectStream) Verify() error

Verify object stream fields.

type PendingObjectsIterator added in v1.87.1 

type PendingObjectsIterator interface {
	Next(ctx context.Context, item *PendingObjectEntry) bool
}

PendingObjectsIterator iterates over a sequence of PendingObjectEntry items.

type Piece 

type Piece struct {
	Number      uint16
	StorageNode storj.NodeID
}

Piece defines information for a segment piece.

type Pieces 

type Pieces []Piece

Pieces defines information for pieces.

func (Pieces) Add added in v1.36.1 

func (p Pieces) Add(piecesToAdd Pieces) (Pieces, error)

Add adds the specified pieces and returns the updated Pieces.

func (Pieces) Equal 

func (p Pieces) Equal(pieces Pieces) bool

Equal checks if Pieces structures are equal.

func (Pieces) FindByNum added in v1.67.1 

func (p Pieces) FindByNum(pieceNum int) (_ Piece, found bool)

FindByNum finds a piece among the Pieces with the given piece number. If no such piece is found, `found` will be returned false.

func (Pieces) Len 

func (p Pieces) Len() int

Len is the number of pieces.

func (Pieces) Less 

func (p Pieces) Less(i, j int) bool

Less reports whether the piece with index i should sort before the piece with index j.

func (Pieces) Remove added in v1.36.1 

func (p Pieces) Remove(piecesToRemove Pieces) (Pieces, error)

Remove removes the specified pieces from the original pieces and returns the updated Pieces.

func (*Pieces) Scan 

func (pieces *Pieces) Scan(value interface{}) error

Scan implements sql.Scanner interface.

func (Pieces) Swap 

func (p Pieces) Swap(i, j int)

Swap swaps the pieces with indexes i and j.

func (Pieces) Update added in v1.36.1 

func (p Pieces) Update(piecesToAdd, piecesToRemove Pieces) (Pieces, error)

Update adds piecesToAdd pieces and removes piecesToRemove pieces from the original pieces struct and returns the updated Pieces.

It removes the piecesToRemove only if all piece number, node id match.

When adding a piece, it checks if the piece already exists using the piece Number If a piece already exists, it returns an empty pieces struct and an error.

func (Pieces) Value 

func (pieces Pieces) Value() (driver.Value, error)

Value implements sql/driver.Valuer interface.

func (Pieces) Verify 

func (p Pieces) Verify() error

Verify verifies pieces.

type PostgresAdapter added in v1.102.2 

type PostgresAdapter struct {
	// contains filtered or unexported fields
}

PostgresAdapter uses Cockroach related SQL queries.

func (*PostgresAdapter) BeginObjectExactVersion added in v1.136.2 

func (p *PostgresAdapter) BeginObjectExactVersion(ctx context.Context, opts BeginObjectExactVersion, object *Object) error

BeginObjectExactVersion implements Adapter.

func (*PostgresAdapter) BeginObjectNextVersion added in v1.102.2 

func (p *PostgresAdapter) BeginObjectNextVersion(ctx context.Context, opts BeginObjectNextVersion, object *Object) error

BeginObjectNextVersion implements Adapter.

func (*PostgresAdapter) BucketEmpty added in v1.106.1 

func (p *PostgresAdapter) BucketEmpty(ctx context.Context, opts BucketEmpty) (empty bool, err error)

BucketEmpty returns true if bucket does not contain objects (pending or committed). This method doesn't check bucket existence.

func (*PostgresAdapter) CheckSegmentPiecesAlteration added in v1.132.5 

func (p *PostgresAdapter) CheckSegmentPiecesAlteration(ctx context.Context, streamID uuid.UUID, position SegmentPosition, aliasPieces AliasPieces) (altered bool, err error)

CheckSegmentPiecesAlteration checks if a segment exists and if its pieces match the provided alias pieces. It returns true if pieces don't match, otherwise false. The comparison is done at the database level for efficiency.

func (*PostgresAdapter) CheckVersion added in v1.112.2 

func (p *PostgresAdapter) CheckVersion(ctx context.Context) error

CheckVersion checks the database is the correct version.

func (*PostgresAdapter) CollectBucketTallies added in v1.107.1 

func (p *PostgresAdapter) CollectBucketTallies(ctx context.Context, opts CollectBucketTallies) (result []BucketTally, err error)

CollectBucketTallies collect limited bucket tallies from given bucket locations.

func (*PostgresAdapter) CommitInlineObject added in v1.148.3 

func (p *PostgresAdapter) CommitInlineObject(ctx context.Context, opts CommitInlineObject) (object Object, err error)

CommitInlineObject adds full inline object to the database.

func (*PostgresAdapter) CommitInlineSegment added in v1.104.1 

func (p *PostgresAdapter) CommitInlineSegment(ctx context.Context, opts CommitInlineSegment) (err error)

CommitInlineSegment commits inline segment to the database.

func (*PostgresAdapter) CommitObject added in v1.148.3 

func (p *PostgresAdapter) CommitObject(ctx context.Context, opts CommitObject) (object Object, err error)

CommitObject adds a pending object to the database.

func (*PostgresAdapter) CommitPendingObjectSegment added in v1.104.1 

func (p *PostgresAdapter) CommitPendingObjectSegment(ctx context.Context, opts CommitSegment, aliasPieces AliasPieces) (err error)

CommitPendingObjectSegment commits segment to the database.

func (*PostgresAdapter) Config added in v1.148.3 

func (p *PostgresAdapter) Config() *Config

Config returns the metabase configuration.

func (*PostgresAdapter) CountSegments added in v1.123.4 

func (p *PostgresAdapter) CountSegments(ctx context.Context, checkTimestamp time.Time) (result int64, err error)

CountSegments returns the number of segments in the segments table.

func (*PostgresAdapter) DeleteAllBucketObjects added in v1.116.3 

func (p *PostgresAdapter) DeleteAllBucketObjects(ctx context.Context, opts DeleteAllBucketObjects) (totalDeletedObjects, totalDeletedSegments int64, err error)

DeleteAllBucketObjects deletes objects in the specified bucket in batches of opts.BatchSize number of objects.

func (*PostgresAdapter) DeleteInactiveObjectsAndSegments added in v1.106.1 

func (p *PostgresAdapter) DeleteInactiveObjectsAndSegments(ctx context.Context, objects []ObjectStream, opts DeleteZombieObjects) (objectsDeleted, segmentsDeleted int64, err error)

DeleteInactiveObjectsAndSegments deletes inactive objects and associated segments.

func (*PostgresAdapter) DeleteObjectExactVersion added in v1.106.1 

func (p *PostgresAdapter) DeleteObjectExactVersion(ctx context.Context, opts DeleteObjectExactVersion) (DeleteObjectResult, error)

DeleteObjectExactVersion deletes an exact object version.

func (*PostgresAdapter) DeleteObjectLastCommittedPlain added in v1.106.1 

func (p *PostgresAdapter) DeleteObjectLastCommittedPlain(ctx context.Context, opts DeleteObjectLastCommitted) (result DeleteObjectResult, err error)

DeleteObjectLastCommittedPlain deletes an object last committed version when opts.Suspended and opts.Versioned are both false.

func (*PostgresAdapter) DeleteObjectLastCommittedVersioned added in v1.106.1 

func (p *PostgresAdapter) DeleteObjectLastCommittedVersioned(ctx context.Context, opts DeleteObjectLastCommitted, deleterMarkerStreamID uuid.UUID) (result DeleteObjectResult, err error)

DeleteObjectLastCommittedVersioned deletes an object last committed version when opts.Versioned is true.

func (*PostgresAdapter) DeleteObjectsAndSegmentsNoVerify added in v1.119.2 

func (p *PostgresAdapter) DeleteObjectsAndSegmentsNoVerify(ctx context.Context, objects []ObjectStream) (objectsDeleted, segmentsDeleted int64, err error)

DeleteObjectsAndSegmentsNoVerify deletes expired objects and associated segments.

func (*PostgresAdapter) DeletePendingObject added in v1.106.1 

func (p *PostgresAdapter) DeletePendingObject(ctx context.Context, opts DeletePendingObject) (result DeleteObjectResult, err error)

DeletePendingObject deletes a pending object with specified version and streamID.

func (*PostgresAdapter) EnsureNodeAliases added in v1.103.2 

func (p *PostgresAdapter) EnsureNodeAliases(ctx context.Context, opts EnsureNodeAliases) (err error)

EnsureNodeAliases implements Adapter.

func (*PostgresAdapter) GetLatestObjectLastSegment added in v1.106.1 

func (p *PostgresAdapter) GetLatestObjectLastSegment(ctx context.Context, opts GetLatestObjectLastSegment) (segment Segment, aliasPieces AliasPieces, err error)

GetLatestObjectLastSegment returns an object last segment information.

func (*PostgresAdapter) GetNodeAliasEntries added in v1.108.1 

func (p *PostgresAdapter) GetNodeAliasEntries(ctx context.Context, opts GetNodeAliasEntries) (_ []NodeAliasEntry, err error)

GetNodeAliasEntries implements Adapter.

func (*PostgresAdapter) GetObjectExactVersion added in v1.106.1 

func (p *PostgresAdapter) GetObjectExactVersion(ctx context.Context, opts GetObjectExactVersion) (_ Object, err error)

GetObjectExactVersion returns object information for exact version.

func (*PostgresAdapter) GetObjectExactVersionLegalHold added in v1.113.1 

func (p *PostgresAdapter) GetObjectExactVersionLegalHold(ctx context.Context, opts GetObjectExactVersionLegalHold) (_ bool, err error)

GetObjectExactVersionLegalHold returns the legal hold configuration of an exact version of an object.

func (*PostgresAdapter) GetObjectExactVersionRetention added in v1.110.1 

func (p *PostgresAdapter) GetObjectExactVersionRetention(ctx context.Context, opts GetObjectExactVersionRetention) (_ Retention, err error)

GetObjectExactVersionRetention returns the retention configuration of an exact version of an object.

func (*PostgresAdapter) GetObjectLastCommitted added in v1.102.2 

func (p *PostgresAdapter) GetObjectLastCommitted(ctx context.Context, opts GetObjectLastCommitted) (object Object, err error)

GetObjectLastCommitted implements Adapter.

func (*PostgresAdapter) GetObjectLastCommittedLegalHold added in v1.113.1 

func (p *PostgresAdapter) GetObjectLastCommittedLegalHold(ctx context.Context, opts GetObjectLastCommittedLegalHold) (_ bool, err error)

GetObjectLastCommittedLegalHold returns the legal hold configuration of the most recently committed version of an object.

func (*PostgresAdapter) GetObjectLastCommittedRetention added in v1.110.1 

func (p *PostgresAdapter) GetObjectLastCommittedRetention(ctx context.Context, opts GetObjectLastCommittedRetention) (_ Retention, err error)

GetObjectLastCommittedRetention returns the retention configuration of the most recently committed version of an object.

func (*PostgresAdapter) GetSegmentByPosition added in v1.106.1 

func (p *PostgresAdapter) GetSegmentByPosition(ctx context.Context, opts GetSegmentByPosition) (segment Segment, aliasPieces AliasPieces, err error)

GetSegmentByPosition returns information about segment on the specified position.

func (*PostgresAdapter) GetSegmentByPositionForAudit added in v1.131.3 

func (p *PostgresAdapter) GetSegmentByPositionForAudit(
	ctx context.Context, opts GetSegmentByPosition,
) (segment SegmentForAudit, aliasPieces AliasPieces, err error)

GetSegmentByPositionForAudit returns information about segment on the specified position for the audit functionality.

func (*PostgresAdapter) GetSegmentByPositionForRepair added in v1.131.3 

func (p *PostgresAdapter) GetSegmentByPositionForRepair(
	ctx context.Context, opts GetSegmentByPosition,
) (segment SegmentForRepair, aliasPieces AliasPieces, err error)

GetSegmentByPositionForRepair returns information about segment on the specified position for the repair functionality.

func (*PostgresAdapter) GetSegmentPositionsAndKeys added in v1.106.1 

func (p *PostgresAdapter) GetSegmentPositionsAndKeys(ctx context.Context, streamID uuid.UUID) (keysNonces []EncryptedKeyAndNonce, err error)

GetSegmentPositionsAndKeys fetches the Position, EncryptedKeyNonce, and EncryptedKey for all segments in the db for the given stream ID, ordered by position.

func (*PostgresAdapter) GetStreamPieceCountByAlias added in v1.107.1 

func (p *PostgresAdapter) GetStreamPieceCountByAlias(ctx context.Context, opts GetStreamPieceCountByNodeID) (result map[NodeAlias]int64, err error)

GetStreamPieceCountByAlias returns piece count by node alias.

func (*PostgresAdapter) GetTableStats added in v1.104.1 

func (p *PostgresAdapter) GetTableStats(ctx context.Context, opts GetTableStats) (result TableStats, err error)

GetTableStats implements Adapter.

func (*PostgresAdapter) Implementation added in v1.112.2 

func (p *PostgresAdapter) Implementation() dbutil.Implementation

Implementation returns the dbutil.Implementation code for this adapter.

func (*PostgresAdapter) IterateExpiredObjects added in v1.122.1 

func (p *PostgresAdapter) IterateExpiredObjects(ctx context.Context, opts DeleteExpiredObjects, process func(context.Context, []ObjectStream) error) (err error)

IterateExpiredObjects iterates over all expired objects that expired before opts.ExpiredBefore and calls process with at most opts.BatchSize objects.

func (*PostgresAdapter) IterateLoopSegments added in v1.103.2 

func (p *PostgresAdapter) IterateLoopSegments(ctx context.Context, aliasCache *NodeAliasCache, opts IterateLoopSegments, fn func(context.Context, LoopSegmentsIterator) error) (err error)

IterateLoopSegments implements Adapter.

func (*PostgresAdapter) IterateZombieObjects added in v1.122.1 

func (p *PostgresAdapter) IterateZombieObjects(ctx context.Context, opts DeleteZombieObjects, process func(context.Context, []ObjectStream) error) (err error)

IterateZombieObjects iterates over all zombie objects and calls process with at most opts.BatchSize objects.

func (*PostgresAdapter) ListBucketStreamIDs added in v1.130.1 

func (p *PostgresAdapter) ListBucketStreamIDs(ctx context.Context, opts ListBucketStreamIDs, process func(ctx context.Context, streamIDs []uuid.UUID) error) error

ListBucketStreamIDs lists the streamIDs from a bucket.

func (*PostgresAdapter) ListNodeAliases added in v1.103.2 

func (p *PostgresAdapter) ListNodeAliases(ctx context.Context) (_ []NodeAliasEntry, err error)

ListNodeAliases implements Adapter.

func (*PostgresAdapter) ListObjects added in v1.106.1 

func (p *PostgresAdapter) ListObjects(ctx context.Context, opts ListObjects) (result ListObjectsResult, err error)

ListObjects lists objects.

func (*PostgresAdapter) ListSegments added in v1.106.1 

func (p *PostgresAdapter) ListSegments(ctx context.Context, opts ListSegments, aliasCache *NodeAliasCache) (result ListSegmentsResult, err error)

ListSegments lists specified stream segments.

func (*PostgresAdapter) ListStreamPositions added in v1.106.1 

func (p *PostgresAdapter) ListStreamPositions(ctx context.Context, opts ListStreamPositions) (result ListStreamPositionsResult, err error)

ListStreamPositions lists specified stream segment positions.

func (*PostgresAdapter) ListVerifySegments added in v1.107.1 

func (p *PostgresAdapter) ListVerifySegments(ctx context.Context, opts ListVerifySegments) (segments []VerifySegment, err error)

ListVerifySegments lists the segments in a specified stream.

func (*PostgresAdapter) MigrateToLatest added in v1.112.2 

func (p *PostgresAdapter) MigrateToLatest(ctx context.Context) error

MigrateToLatest migrates database to the latest version.

func (*PostgresAdapter) Name added in v1.106.1 

func (p *PostgresAdapter) Name() string

Name returns the name of the adapter.

func (*PostgresAdapter) Now added in v1.107.1 

func (p *PostgresAdapter) Now(ctx context.Context) (time.Time, error)

Now returns the current time according to the database.

func (*PostgresAdapter) PendingObjectExists added in v1.104.1 

func (p *PostgresAdapter) PendingObjectExists(ctx context.Context, opts BeginSegment) (exists bool, err error)

PendingObjectExists checks whether an object already exists.

func (*PostgresAdapter) Ping added in v1.107.1 

func (p *PostgresAdapter) Ping(ctx context.Context) error

Ping checks whether connection has been established.

func (*PostgresAdapter) PostgresMigration added in v1.112.2 

func (p *PostgresAdapter) PostgresMigration() *migrate.Migration

PostgresMigration returns steps needed for migrating postgres database.

func (*PostgresAdapter) SetObjectExactVersionLegalHold added in v1.113.1 

func (p *PostgresAdapter) SetObjectExactVersionLegalHold(ctx context.Context, opts SetObjectExactVersionLegalHold) (err error)

SetObjectExactVersionLegalHold sets the legal hold configuration of an exact version of an object.

func (*PostgresAdapter) SetObjectExactVersionRetention added in v1.110.1 

func (p *PostgresAdapter) SetObjectExactVersionRetention(ctx context.Context, opts SetObjectExactVersionRetention) (err error)

SetObjectExactVersionRetention sets the retention configuration of an exact version of an object.

func (*PostgresAdapter) SetObjectLastCommittedLegalHold added in v1.113.1 

func (p *PostgresAdapter) SetObjectLastCommittedLegalHold(ctx context.Context, opts SetObjectLastCommittedLegalHold) (err error)

SetObjectLastCommittedLegalHold sets the legal hold configuration of the most recently committed version of an object.

func (*PostgresAdapter) SetObjectLastCommittedRetention added in v1.110.1 

func (p *PostgresAdapter) SetObjectLastCommittedRetention(ctx context.Context, opts SetObjectLastCommittedRetention) (err error)

SetObjectLastCommittedRetention sets the retention configuration of the most recently committed version of an object.

func (*PostgresAdapter) TestMigrateToLatest added in v1.110.1 

func (p *PostgresAdapter) TestMigrateToLatest(ctx context.Context) error

TestMigrateToLatest creates a database and applies all the migration for test purposes.

func (*PostgresAdapter) TestingBatchInsertObjects added in v1.106.1 

func (p *PostgresAdapter) TestingBatchInsertObjects(ctx context.Context, objects []RawObject) (err error)

TestingBatchInsertObjects batch inserts objects for testing.

func (*PostgresAdapter) TestingBatchInsertSegments added in v1.102.2 

func (p *PostgresAdapter) TestingBatchInsertSegments(ctx context.Context, aliasCache *NodeAliasCache, segments []RawSegment) (err error)

TestingBatchInsertSegments implements postgres adapter.

func (*PostgresAdapter) TestingDeleteAll added in v1.103.2 

func (p *PostgresAdapter) TestingDeleteAll(ctx context.Context) (err error)

TestingDeleteAll implements Adapter.

func (*PostgresAdapter) TestingGetAllObjects added in v1.104.1 

func (p *PostgresAdapter) TestingGetAllObjects(ctx context.Context) (_ []RawObject, err error)

TestingGetAllObjects returns the state of the database.

func (*PostgresAdapter) TestingGetAllSegments added in v1.103.2 

func (p *PostgresAdapter) TestingGetAllSegments(ctx context.Context, aliasCache *NodeAliasCache) (_ []RawSegment, err error)

TestingGetAllSegments implements Adapter.

func (*PostgresAdapter) TestingSetObjectCreatedAt added in v1.147.4 

func (p *PostgresAdapter) TestingSetObjectCreatedAt(ctx context.Context, object ObjectStream, createdAt time.Time) (rowsAffected int64, err error)

TestingSetObjectCreatedAt sets the created_at of the object to the given value in tests.

func (*PostgresAdapter) TestingSetObjectVersion added in v1.112.2 

func (p *PostgresAdapter) TestingSetObjectVersion(ctx context.Context, object ObjectStream, randomVersion Version) (rowsAffected int64, err error)

TestingSetObjectVersion sets the version of the object to the given value.

func (*PostgresAdapter) TestingSetPlacementAllSegments added in v1.112.2 

func (p *PostgresAdapter) TestingSetPlacementAllSegments(ctx context.Context, placement storj.PlacementConstraint) (err error)

TestingSetPlacementAllSegments sets the placement of all segments to the given value.

func (*PostgresAdapter) UncoordinatedDeleteAllBucketObjects added in v1.132.2 

func (p *PostgresAdapter) UncoordinatedDeleteAllBucketObjects(ctx context.Context, opts UncoordinatedDeleteAllBucketObjects) (totalDeletedObjects, totalDeletedSegments int64, err error)

UncoordinatedDeleteAllBucketObjects deletes objects in the specified bucket in batches of opts.BatchSize number of objects.

func (*PostgresAdapter) UnderlyingDB added in v1.107.1 

func (p *PostgresAdapter) UnderlyingDB() tagsql.DB

UnderlyingDB returns a handle to the underlying DB.

func (*PostgresAdapter) UpdateObjectLastCommittedMetadata added in v1.106.1 

func (p *PostgresAdapter) UpdateObjectLastCommittedMetadata(ctx context.Context, opts UpdateObjectLastCommittedMetadata) (affected int64, err error)

UpdateObjectLastCommittedMetadata updates an object metadata.

func (*PostgresAdapter) UpdateSegmentPieces added in v1.106.1 

func (p *PostgresAdapter) UpdateSegmentPieces(ctx context.Context, opts UpdateSegmentPieces, oldPieces, newPieces AliasPieces) (resultPieces AliasPieces, err error)

UpdateSegmentPieces updates pieces for specified segment, if pieces matches oldPieces.

func (*PostgresAdapter) UpdateTableStats added in v1.107.1 

func (p *PostgresAdapter) UpdateTableStats(ctx context.Context) error

UpdateTableStats forces an update of table statistics. Probably useful mostly in test scenarios.

func (*PostgresAdapter) WithTx added in v1.104.1 

func (p *PostgresAdapter) WithTx(ctx context.Context, opts TransactionOptions, f func(context.Context, TransactionAdapter) error) error

WithTx provides a TransactionAdapter for the context of a database transaction.

type PrecommitInfo added in v1.140.3 

type PrecommitInfo struct {
	ObjectStream

	// TimestampVersion is used for timestamp versioning.
	//
	// This is used when timestamp versioning is enabled and we need to change version.
	// We request it from the database to have a consistent source of time.
	TimestampVersion Version
	// HighestVersion is the highest object version in the database.
	//
	// This is needed to determine whether the current pending object is the
	// latest and we can avoid changing the primary key. If it's not the newest
	// we can use it to generate the new version, when not using timestamp versioning.
	HighestVersion Version
	// Pending contains all the fields for the object to be committed.
	// This is used to reinsert the object when primary key cannot be changed.
	//
	// Encrypted fields are also necessary to verify when updating encrypted metadata.
	//
	// TODO: the amount of data transferred can probably reduced by doing a conditional
	// query.
	Pending *PrecommitPendingObject
	// Segments contains all the segments for the given object.
	Segments []PrecommitSegment
	// HighestVisible returns the status of the highest version that's either committed
	// or a delete marker.
	//
	// This is used to handle "IfNoneMatch" query. We need to know whether
	// the we consider the object to exist or not.
	HighestVisible ObjectStatus
	// Unversioned is the unversioned object at the given location. It is
	// returned when params.Unversioned or params.FullUnversioned is true.
	//
	// This is used to delete the previous unversioned object at the location,
	// which ensures that there's only one unversioned object at a given location.
	Unversioned *PrecommitUnversionedObject

	// FullUnversioned is the unversioned object at the given location.
	// It is returned when params.FullUnversioned is true.
	FullUnversioned *RawObject
}

PrecommitInfo is the information necessary for committing objects.

type PrecommitPendingObject added in v1.140.3 

type PrecommitPendingObject struct {
	CreatedAt                     time.Time                  `spanner:"created_at"`
	ExpiresAt                     *time.Time                 `spanner:"expires_at"`
	EncryptedMetadata             []byte                     `spanner:"encrypted_metadata"`
	EncryptedMetadataNonce        []byte                     `spanner:"encrypted_metadata_nonce"`
	EncryptedMetadataEncryptedKey []byte                     `spanner:"encrypted_metadata_encrypted_key"`
	EncryptedETag                 []byte                     `spanner:"encrypted_etag"`
	Encryption                    storj.EncryptionParameters `spanner:"encryption"`
	RetentionMode                 RetentionMode              `spanner:"retention_mode"`
	RetainUntil                   spanner.NullTime           `spanner:"retain_until"`
}

PrecommitPendingObject is information about the object to be committed.

type PrecommitQuery added in v1.140.3 

type PrecommitQuery struct {
	ObjectStream
	// Pending returns the pending object and segments at the location. Precommit returns an error when it does not exist.
	Pending bool
	// ExcludeFromPending contains fields to exclude from the pending object.
	ExcludeFromPending ExcludeFromPending
	// Unversioned returns the unversioned object at the location.
	Unversioned bool
	// FullUnversioned returns all properties of the unversioned object at the location.
	FullUnversioned bool
	// HighestVisible returns the highest committed object or delete marker at the location.
	HighestVisible bool
}

PrecommitQuery is used for querying precommit info.

type PrecommitSegment added in v1.140.3 

type PrecommitSegment struct {
	Position      SegmentPosition
	EncryptedSize int32
	PlainOffset   int64
	PlainSize     int32
}

PrecommitSegment is segment state before committing the object.

type PrecommitUnversionedObject added in v1.140.3 

type PrecommitUnversionedObject struct {
	Version       Version          `spanner:"version"`
	StreamID      uuid.UUID        `spanner:"stream_id"`
	RetentionMode RetentionMode    `spanner:"retention_mode"`
	RetainUntil   spanner.NullTime `spanner:"retain_until"`
}

PrecommitUnversionedObject is information necessary to delete unversioned object at a given location.

func PrecommitUnversionedObjectFromObject added in v1.142.7 

func PrecommitUnversionedObjectFromObject(obj *RawObject) *PrecommitUnversionedObject

PrecommitUnversionedObjectFromObject creates a unversioned object from raw object.

type RawCopy added in v1.50.1 

type RawCopy struct {
	StreamID         uuid.UUID
	AncestorStreamID uuid.UUID
}

RawCopy contains a copy that is stored in the database.

type RawObject 

type RawObject struct {
	ObjectStream

	CreatedAt time.Time
	ExpiresAt *time.Time

	Status       ObjectStatus
	SegmentCount int32

	EncryptedUserData

	// TotalPlainSize is 0 for a migrated object.
	TotalPlainSize     int64
	TotalEncryptedSize int64
	// FixedSegmentSize is 0 for a migrated object.
	FixedSegmentSize int32

	Encryption storj.EncryptionParameters

	// ZombieDeletionDeadline defines when the pending raw object should be deleted from the database.
	// This is as a safeguard against objects that failed to upload and the client has not indicated
	// whether they want to continue uploading or delete the already uploaded data.
	ZombieDeletionDeadline *time.Time

	Retention Retention
	LegalHold bool
}

RawObject defines the full object that is stored in the database. It should be rarely used directly.

type RawSegment 

type RawSegment struct {
	StreamID uuid.UUID
	Position SegmentPosition

	CreatedAt  time.Time // non-nillable
	RepairedAt *time.Time
	ExpiresAt  *time.Time

	RootPieceID       storj.PieceID
	EncryptedKeyNonce []byte
	EncryptedKey      []byte

	EncryptedSize int32 // size of the whole segment (not a piece)
	// PlainSize is 0 for a migrated object.
	PlainSize int32
	// PlainOffset is 0 for a migrated object.
	PlainOffset   int64
	EncryptedETag []byte

	Redundancy storj.RedundancyScheme

	InlineData []byte
	Pieces     Pieces

	Placement storj.PlacementConstraint
}

RawSegment defines the full segment that is stored in the database. It should be rarely used directly.

type RawState 

type RawState struct {
	Objects  []RawObject
	Segments []RawSegment
}

RawState contains full state of a table.

type Retention added in v1.109.1 

type Retention struct {
	Mode        storj.RetentionMode
	RetainUntil time.Time
}

Retention represents an object version's Object Lock retention configuration.

func (*Retention) Active added in v1.110.1 

func (r *Retention) Active(now time.Time) bool

Active returns whether the retention configuration is enabled and active as of the given time.

func (*Retention) ActiveNow added in v1.112.2 

func (r *Retention) ActiveNow() bool

ActiveNow returns whether the retention configuration is enabled and active as of the current time.

func (*Retention) Enabled added in v1.109.1 

func (r *Retention) Enabled() bool

Enabled returns whether the retention configuration is enabled.

func (*Retention) Verify added in v1.109.1 

func (r *Retention) Verify() error

Verify verifies the retention configuration.

type RetentionMode added in v1.140.3 

type RetentionMode struct {
	Mode      storj.RetentionMode
	LegalHold bool
}

RetentionMode implements scanning for retention_mode column.

func (*RetentionMode) DecodeSpanner added in v1.140.3 

func (r *RetentionMode) DecodeSpanner(val interface{}) error

DecodeSpanner implements the spanner.Decoder interface.

func (RetentionMode) EncodeSpanner added in v1.140.3 

func (r RetentionMode) EncodeSpanner() (interface{}, error)

EncodeSpanner implements the spanner.Encoder interface.

func (*RetentionMode) Scan added in v1.140.3 

func (r *RetentionMode) Scan(val interface{}) error

Scan implements the sql.Scanner interface.

func (RetentionMode) Value added in v1.140.3 

func (r RetentionMode) Value() (driver.Value, error)

Value implements the sql/driver.Valuer interface.

type Segment 

type Segment RawSegment

Segment segment metadata. TODO define separated struct.

func (Segment) Expired added in v1.34.1 

func (s Segment) Expired(now time.Time) bool

Expired checks if segment is expired relative to now.

func (Segment) Inline 

func (s Segment) Inline() bool

Inline returns true if segment is inline.

func (Segment) PieceSize added in v1.74.1 

func (s Segment) PieceSize() int64

PieceSize returns calculated piece size for segment.

type SegmentForAudit added in v1.131.3 

type SegmentForAudit SegmentForRepair

SegmentForAudit defines the segment data required for the audit functionality.

func (SegmentForAudit) Expired added in v1.131.3 

func (s SegmentForAudit) Expired(now time.Time) bool

Expired checks if segment is expired relative to now.

func (SegmentForAudit) PieceSize added in v1.131.3 

func (s SegmentForAudit) PieceSize() int64

PieceSize returns calculated piece size for segment.

type SegmentForRepair added in v1.131.3 

type SegmentForRepair struct {
	StreamID uuid.UUID
	Position SegmentPosition

	CreatedAt  time.Time // non-nillable
	RepairedAt *time.Time
	ExpiresAt  *time.Time

	RootPieceID   storj.PieceID
	EncryptedSize int32 // size of the whole segment (not a piece)
	Redundancy    storj.RedundancyScheme
	Pieces        Pieces
	Placement     storj.PlacementConstraint
}

SegmentForRepair defines the segment data required for the repair functionality.

func (SegmentForRepair) Expired added in v1.131.3 

func (s SegmentForRepair) Expired(now time.Time) bool

Expired checks if segment is expired relative to now.

func (SegmentForRepair) Inline added in v1.131.3 

func (s SegmentForRepair) Inline() bool

Inline returns true if segment is inline.

func (SegmentForRepair) PieceSize added in v1.131.3 

func (s SegmentForRepair) PieceSize() int64

PieceSize returns calculated piece size for segment.

type SegmentKey 

type SegmentKey []byte

SegmentKey is an encoded metainfo key. This is used as the key in pointerdb key-value store.

type SegmentLocation 

type SegmentLocation struct {
	ProjectID  uuid.UUID
	BucketName BucketName
	ObjectKey  ObjectKey
	Position   SegmentPosition
}

SegmentLocation is decoded segment key information.

func ParseSegmentKey 

func ParseSegmentKey(encoded SegmentKey) (SegmentLocation, error)

ParseSegmentKey parses an segment key into segment location.

func (SegmentLocation) Bucket 

func (seg SegmentLocation) Bucket() BucketLocation

Bucket returns bucket location this segment belongs to.

func (SegmentLocation) Encode 

func (seg SegmentLocation) Encode() SegmentKey

Encode converts segment location into a segment key.

func (SegmentLocation) Object 

func (seg SegmentLocation) Object() ObjectLocation

Object returns the object location associated with this segment location.

func (SegmentLocation) Verify 

func (seg SegmentLocation) Verify() error

Verify segment location fields.

type SegmentPosition 

type SegmentPosition struct {
	Part  uint32
	Index uint32
}

SegmentPosition is segment part and index combined.

func SegmentPositionFromEncoded 

func SegmentPositionFromEncoded(v uint64) SegmentPosition

SegmentPositionFromEncoded decodes an uint64 into a SegmentPosition.

func (*SegmentPosition) DecodeSpanner added in v1.103.2 

func (pos *SegmentPosition) DecodeSpanner(val any) (err error)

DecodeSpanner implements spanner.Decoder.

func (SegmentPosition) Encode 

func (pos SegmentPosition) Encode() uint64

Encode encodes a segment position into an uint64, that can be stored in a database.

func (SegmentPosition) EncodeSpanner added in v1.103.2 

func (pos SegmentPosition) EncodeSpanner() (any, error)

EncodeSpanner implements spanner.Encoder.

func (SegmentPosition) Less 

func (pos SegmentPosition) Less(b SegmentPosition) bool

Less returns whether pos should before b.

func (*SegmentPosition) Scan 

func (params *SegmentPosition) Scan(value interface{}) error

Scan implements sql.Scanner interface.

func (SegmentPosition) Value 

func (params SegmentPosition) Value() (driver.Value, error)

Value implements sql/driver.Valuer interface.

type SegmentPositionInfo 

type SegmentPositionInfo struct {
	Position SegmentPosition
	// PlainSize is 0 for a migrated object.
	PlainSize int32
	// PlainOffset is 0 for a migrated object.
	PlainOffset       int64
	CreatedAt         *time.Time // TODO: make it non-nilable after we migrate all existing segments to have creation time
	EncryptedETag     []byte
	EncryptedKeyNonce []byte
	EncryptedKey      []byte
}

SegmentPositionInfo contains information for segment position.

type SegmentsStats added in v1.123.4 

type SegmentsStats struct {
	SegmentCount           int64
	PerAdapterSegmentCount []int64
}

SegmentsStats contains information about the segments table.

type SetObjectExactVersionLegalHold added in v1.113.1 

type SetObjectExactVersionLegalHold struct {
	ObjectLocation
	Version Version

	Enabled bool
}

SetObjectExactVersionLegalHold contains arguments necessary for setting the legal hold configuration of an exact version of an object.

func (*SetObjectExactVersionLegalHold) Verify added in v1.113.1 

func (opts *SetObjectExactVersionLegalHold) Verify() error

Verify verifies the request fields.

type SetObjectExactVersionRetention added in v1.110.1 

type SetObjectExactVersionRetention struct {
	ObjectLocation
	Version Version

	Retention        Retention
	BypassGovernance bool
}

SetObjectExactVersionRetention contains arguments necessary for setting the retention configuration of an exact version of an object.

func (*SetObjectExactVersionRetention) Verify added in v1.110.1 

func (opts *SetObjectExactVersionRetention) Verify() (err error)

Verify verifies the request fields.

type SetObjectLastCommittedLegalHold added in v1.113.1 

type SetObjectLastCommittedLegalHold struct {
	ObjectLocation

	Enabled bool
}

SetObjectLastCommittedLegalHold contains arguments necessary for setting the legal hold configuration of the most recently committed version of an object.

func (SetObjectLastCommittedLegalHold) Verify added in v1.113.1 

func (opts SetObjectLastCommittedLegalHold) Verify() error

Verify verifies the request fields.

type SetObjectLastCommittedRetention added in v1.110.1 

type SetObjectLastCommittedRetention struct {
	ObjectLocation

	Retention        Retention
	BypassGovernance bool
}

SetObjectLastCommittedRetention contains arguments necessary for setting the retention configuration of the most recently committed version of an object.

func (SetObjectLastCommittedRetention) Verify added in v1.110.1 

func (opts SetObjectLastCommittedRetention) Verify() (err error)

Verify verifies the request fields.

type Shard added in v1.148.3 

type Shard interface {
	CommitObject(ctx context.Context, opts CommitObject) (object Object, err error)
	CommitInlineObject(ctx context.Context, opts CommitInlineObject) (object Object, err error)
}

Shard represents methods that are specific to a particular database implementation. Right now it contains only method that were fully moved under specific DB implementation Postgres/CRDB or Spanner.

type SpannerAdapter added in v1.103.2 

type SpannerAdapter struct {
	// contains filtered or unexported fields
}

SpannerAdapter implements Adapter for Google Spanner connections..

func NewSpannerAdapter added in v1.103.2 

func NewSpannerAdapter(ctx context.Context, log *zap.Logger, cfg SpannerConfig, config *Config, recorder *flightrecorder.Box) (*SpannerAdapter, error)

NewSpannerAdapter creates a new Spanner adapter.

func (*SpannerAdapter) AddChangeStreamPartition added in v1.141.5 

func (s *SpannerAdapter) AddChangeStreamPartition(ctx context.Context, feedName, childToken string, parentTokens []string, start time.Time) error

AddChangeStreamPartition adds a child partition to the metabase.

func (*SpannerAdapter) BeginObjectExactVersion added in v1.136.2 

func (s *SpannerAdapter) BeginObjectExactVersion(ctx context.Context, opts BeginObjectExactVersion, object *Object) error

BeginObjectExactVersion implements Adapter.

func (*SpannerAdapter) BeginObjectNextVersion added in v1.103.2 

func (s *SpannerAdapter) BeginObjectNextVersion(ctx context.Context, opts BeginObjectNextVersion, object *Object) error

BeginObjectNextVersion implements Adapter.

func (*SpannerAdapter) BucketEmpty added in v1.106.1 

func (s *SpannerAdapter) BucketEmpty(ctx context.Context, opts BucketEmpty) (empty bool, err error)

BucketEmpty returns true if bucket does not contain objects (pending or committed). This method doesn't check bucket existence.

func (*SpannerAdapter) ChangeStreamNoPartitionMetadata added in v1.141.5 

func (s *SpannerAdapter) ChangeStreamNoPartitionMetadata(ctx context.Context, feedName string) (bool, error)

ChangeStreamNoPartitionMetadata checks if the metadata table for the change stream is empty.

func (*SpannerAdapter) CheckSegmentPiecesAlteration added in v1.132.5 

func (s *SpannerAdapter) CheckSegmentPiecesAlteration(ctx context.Context, streamID uuid.UUID, position SegmentPosition, aliasPieces AliasPieces) (altered bool, err error)

CheckSegmentPiecesAlteration checks if a segment exists and if its pieces match the provided alias pieces. It returns true if pieces don't match, otherwise false. The comparison is done at the database level for efficiency.

func (*SpannerAdapter) CheckVersion added in v1.112.2 

func (s *SpannerAdapter) CheckVersion(ctx context.Context) error

CheckVersion checks the database is the correct version.

func (*SpannerAdapter) Close added in v1.103.2 

func (s *SpannerAdapter) Close() error

Close closes the internal client.

func (*SpannerAdapter) CollectBucketTallies added in v1.107.1 

func (s *SpannerAdapter) CollectBucketTallies(ctx context.Context, opts CollectBucketTallies) (result []BucketTally, err error)

CollectBucketTallies collect limited bucket tallies from given bucket locations.

func (*SpannerAdapter) CommitInlineObject added in v1.148.3 

func (s *SpannerAdapter) CommitInlineObject(ctx context.Context, opts CommitInlineObject) (object Object, err error)

CommitInlineObject adds full inline object to the database.

func (*SpannerAdapter) CommitInlineSegment added in v1.104.1 

func (s *SpannerAdapter) CommitInlineSegment(ctx context.Context, opts CommitInlineSegment) (err error)

CommitInlineSegment commits inline segment to the database.

func (*SpannerAdapter) CommitObject added in v1.148.3 

func (s *SpannerAdapter) CommitObject(ctx context.Context, opts CommitObject) (object Object, err error)

CommitObject adds a pending object to the database.

func (*SpannerAdapter) CommitPendingObjectSegment added in v1.104.1 

func (s *SpannerAdapter) CommitPendingObjectSegment(ctx context.Context, opts CommitSegment, aliasPieces AliasPieces) (err error)

CommitPendingObjectSegment commits segment to the database.

func (*SpannerAdapter) Config added in v1.148.3 

func (s *SpannerAdapter) Config() *Config

Config returns the metabase configuration.

func (*SpannerAdapter) CountSegments added in v1.123.4 

func (s *SpannerAdapter) CountSegments(ctx context.Context, checkTimestamp time.Time) (result int64, err error)

CountSegments returns the number of segments in the segments table.

func (*SpannerAdapter) DeleteAllBucketObjects added in v1.116.3 

func (s *SpannerAdapter) DeleteAllBucketObjects(ctx context.Context, opts DeleteAllBucketObjects) (totalDeletedObjects, totalDeletedSegments int64, err error)

DeleteAllBucketObjects deletes objects in the specified bucket in batches of opts.BatchSize number of objects.

func (*SpannerAdapter) DeleteInactiveObjectsAndSegments added in v1.106.1 

func (s *SpannerAdapter) DeleteInactiveObjectsAndSegments(ctx context.Context, objects []ObjectStream, opts DeleteZombieObjects) (objectsDeleted, segmentsDeleted int64, err error)

DeleteInactiveObjectsAndSegments deletes inactive objects and associated segments.

func (*SpannerAdapter) DeleteObjectExactVersion added in v1.106.1 

func (s *SpannerAdapter) DeleteObjectExactVersion(ctx context.Context, opts DeleteObjectExactVersion) (DeleteObjectResult, error)

DeleteObjectExactVersion deletes an exact object version.

func (*SpannerAdapter) DeleteObjectLastCommittedPlain added in v1.106.1 

func (s *SpannerAdapter) DeleteObjectLastCommittedPlain(ctx context.Context, opts DeleteObjectLastCommitted) (DeleteObjectResult, error)

DeleteObjectLastCommittedPlain deletes an object last committed version when opts.Suspended and opts.Versioned are both false.

func (*SpannerAdapter) DeleteObjectLastCommittedVersioned added in v1.106.1 

func (s *SpannerAdapter) DeleteObjectLastCommittedVersioned(ctx context.Context, opts DeleteObjectLastCommitted, deleterMarkerStreamID uuid.UUID) (result DeleteObjectResult, err error)

DeleteObjectLastCommittedVersioned deletes an object last committed version when opts.Versioned is true.

func (*SpannerAdapter) DeleteObjectsAndSegmentsNoVerify added in v1.119.2 

func (s *SpannerAdapter) DeleteObjectsAndSegmentsNoVerify(ctx context.Context, objects []ObjectStream) (objectsDeleted, segmentsDeleted int64, err error)

DeleteObjectsAndSegmentsNoVerify deletes expired objects and associated segments.

The implementation does not do extra verification whether the stream id-s belong or belonged to the objects. So, if the callers supplies objects with incorrect StreamID-s it may end up deleting unrelated segments.

func (*SpannerAdapter) DeletePendingObject added in v1.106.1 

func (s *SpannerAdapter) DeletePendingObject(ctx context.Context, opts DeletePendingObject) (result DeleteObjectResult, err error)

DeletePendingObject deletes a pending object with specified version and streamID.

func (*SpannerAdapter) EnsureNodeAliases added in v1.103.2 

func (s *SpannerAdapter) EnsureNodeAliases(ctx context.Context, opts EnsureNodeAliases) (err error)

EnsureNodeAliases implements Adapter.

func (*SpannerAdapter) GetChangeStreamPartitionsByState added in v1.141.5 

func (s *SpannerAdapter) GetChangeStreamPartitionsByState(ctx context.Context, name string, state changestream.PartitionState) (map[string]time.Time, error)

GetChangeStreamPartitionsByState retrieves change stream partitions by their state from the metabase.

func (*SpannerAdapter) GetLatestObjectLastSegment added in v1.106.1 

func (s *SpannerAdapter) GetLatestObjectLastSegment(ctx context.Context, opts GetLatestObjectLastSegment) (segment Segment, aliasPieces AliasPieces, err error)

GetLatestObjectLastSegment returns an object last segment information.

func (*SpannerAdapter) GetNodeAliasEntries added in v1.108.1 

func (s *SpannerAdapter) GetNodeAliasEntries(ctx context.Context, opts GetNodeAliasEntries) (_ []NodeAliasEntry, err error)

GetNodeAliasEntries implements Adapter.

func (*SpannerAdapter) GetObjectExactVersion added in v1.106.1 

func (s *SpannerAdapter) GetObjectExactVersion(ctx context.Context, opts GetObjectExactVersion) (object Object, err error)

GetObjectExactVersion returns object information for exact version.

func (*SpannerAdapter) GetObjectExactVersionLegalHold added in v1.113.1 

func (s *SpannerAdapter) GetObjectExactVersionLegalHold(ctx context.Context, opts GetObjectExactVersionLegalHold) (_ bool, err error)

GetObjectExactVersionLegalHold returns the legal hold configuration of an exact version of an object.

func (*SpannerAdapter) GetObjectExactVersionRetention added in v1.110.1 

func (s *SpannerAdapter) GetObjectExactVersionRetention(ctx context.Context, opts GetObjectExactVersionRetention) (_ Retention, err error)

GetObjectExactVersionRetention returns the retention configuration of an exact version of an object.

func (*SpannerAdapter) GetObjectLastCommitted added in v1.103.2 

func (s *SpannerAdapter) GetObjectLastCommitted(ctx context.Context, opts GetObjectLastCommitted) (object Object, err error)

GetObjectLastCommitted implements Adapter.

func (*SpannerAdapter) GetObjectLastCommittedLegalHold added in v1.113.1 

func (s *SpannerAdapter) GetObjectLastCommittedLegalHold(ctx context.Context, opts GetObjectLastCommittedLegalHold) (_ bool, err error)

GetObjectLastCommittedLegalHold returns the legal hold configuration of the most recently committed version of an object.

func (*SpannerAdapter) GetObjectLastCommittedRetention added in v1.110.1 

func (s *SpannerAdapter) GetObjectLastCommittedRetention(ctx context.Context, opts GetObjectLastCommittedRetention) (_ Retention, err error)

GetObjectLastCommittedRetention returns the retention configuration of the most recently committed version of an object.

func (*SpannerAdapter) GetSegmentByPosition added in v1.106.1 

func (s *SpannerAdapter) GetSegmentByPosition(ctx context.Context, opts GetSegmentByPosition) (segment Segment, aliasPieces AliasPieces, err error)

GetSegmentByPosition returns information about segment on the specified position.

func (*SpannerAdapter) GetSegmentByPositionForAudit added in v1.131.3 

func (s *SpannerAdapter) GetSegmentByPositionForAudit(
	ctx context.Context, opts GetSegmentByPosition,
) (segment SegmentForAudit, aliasPieces AliasPieces, err error)

GetSegmentByPositionForAudit returns information about segment on the specified position for the audit functionality.

func (*SpannerAdapter) GetSegmentByPositionForRepair added in v1.131.3 

func (s *SpannerAdapter) GetSegmentByPositionForRepair(
	ctx context.Context, opts GetSegmentByPosition,
) (segment SegmentForRepair, aliasPieces AliasPieces, err error)

GetSegmentByPositionForRepair returns information about segment on the specified position for the repair functionality.

func (*SpannerAdapter) GetSegmentPositionsAndKeys added in v1.106.1 

func (s *SpannerAdapter) GetSegmentPositionsAndKeys(ctx context.Context, streamID uuid.UUID) (keysNonces []EncryptedKeyAndNonce, err error)

GetSegmentPositionsAndKeys fetches the Position, EncryptedKeyNonce, and EncryptedKey for all segments in the db for the given stream ID, ordered by position.

func (*SpannerAdapter) GetStreamPieceCountByAlias added in v1.107.1 

func (s *SpannerAdapter) GetStreamPieceCountByAlias(ctx context.Context, opts GetStreamPieceCountByNodeID) (result map[NodeAlias]int64, err error)

GetStreamPieceCountByAlias returns piece count by node alias.

func (*SpannerAdapter) GetTableStats added in v1.104.1 

func (s *SpannerAdapter) GetTableStats(ctx context.Context, opts GetTableStats) (result TableStats, err error)

GetTableStats (will) implement Adapter.

func (*SpannerAdapter) Implementation added in v1.112.2 

func (s *SpannerAdapter) Implementation() dbutil.Implementation

Implementation returns the dbutil.Implementation code for the adapter.

func (*SpannerAdapter) IsEmulator added in v1.139.1 

func (s *SpannerAdapter) IsEmulator() bool

IsEmulator returns true if the underlying DB is spanner emulator

func (*SpannerAdapter) IterateExpiredObjects added in v1.122.1 

func (s *SpannerAdapter) IterateExpiredObjects(ctx context.Context, opts DeleteExpiredObjects, process func(context.Context, []ObjectStream) error) (err error)

IterateExpiredObjects iterates over all expired objects that expired before opts.ExpiredBefore and calls process with at most opts.BatchSize objects.

func (*SpannerAdapter) IterateLoopSegments added in v1.103.2 

func (s *SpannerAdapter) IterateLoopSegments(ctx context.Context, aliasCache *NodeAliasCache, opts IterateLoopSegments, fn func(context.Context, LoopSegmentsIterator) error) (err error)

IterateLoopSegments implements Adapter.

func (*SpannerAdapter) IterateZombieObjects added in v1.122.1 

func (s *SpannerAdapter) IterateZombieObjects(ctx context.Context, opts DeleteZombieObjects, process func(context.Context, []ObjectStream) error) (err error)

IterateZombieObjects iterates over all zombie objects and calls process with at most opts.BatchSize objects.

func (*SpannerAdapter) ListBucketStreamIDs added in v1.130.1 

func (s *SpannerAdapter) ListBucketStreamIDs(ctx context.Context, opts ListBucketStreamIDs, process func(ctx context.Context, streamIDs []uuid.UUID) error) error

ListBucketStreamIDs lists the streamIDs from a bucket.

func (*SpannerAdapter) ListNodeAliases added in v1.103.2 

func (s *SpannerAdapter) ListNodeAliases(ctx context.Context) (aliases []NodeAliasEntry, err error)

ListNodeAliases implements Adapter.

func (*SpannerAdapter) ListObjects added in v1.106.1 

func (s *SpannerAdapter) ListObjects(ctx context.Context, opts ListObjects) (result ListObjectsResult, err error)

ListObjects lists objects.

func (*SpannerAdapter) ListSegments added in v1.106.1 

func (s *SpannerAdapter) ListSegments(ctx context.Context, opts ListSegments, aliasCache *NodeAliasCache) (result ListSegmentsResult, err error)

ListSegments lists specified stream segments.

func (*SpannerAdapter) ListStreamPositions added in v1.106.1 

func (s *SpannerAdapter) ListStreamPositions(ctx context.Context, opts ListStreamPositions) (result ListStreamPositionsResult, err error)

ListStreamPositions lists specified stream segment positions.

func (*SpannerAdapter) ListVerifySegments added in v1.107.1 

func (s *SpannerAdapter) ListVerifySegments(ctx context.Context, opts ListVerifySegments) (segments []VerifySegment, err error)

ListVerifySegments lists the segments in a specified stream.

func (*SpannerAdapter) MigrateToLatest added in v1.112.2 

func (s *SpannerAdapter) MigrateToLatest(ctx context.Context) error

MigrateToLatest migrates database to the latest version.

func (*SpannerAdapter) Name added in v1.106.1 

func (s *SpannerAdapter) Name() string

Name returns the name of the adapter.

func (*SpannerAdapter) Now added in v1.107.1 

func (s *SpannerAdapter) Now(ctx context.Context) (time.Time, error)

Now returns the current time according to the database.

func (*SpannerAdapter) PendingObjectExists added in v1.104.1 

func (s *SpannerAdapter) PendingObjectExists(ctx context.Context, opts BeginSegment) (exists bool, err error)

PendingObjectExists checks whether an object already exists.

func (*SpannerAdapter) Ping added in v1.107.1 

func (s *SpannerAdapter) Ping(ctx context.Context) error

Ping checks whether connection has been established.

func (*SpannerAdapter) ReadChangeStreamPartition added in v1.141.5 

func (s *SpannerAdapter) ReadChangeStreamPartition(ctx context.Context, name string, partitionToken string, from time.Time, callback func(record changestream.ChangeRecord) error) error

ReadChangeStreamPartition reads records from a change stream partition and processes records via callback.

func (*SpannerAdapter) ScheduleChangeStreamPartitions added in v1.141.5 

func (s *SpannerAdapter) ScheduleChangeStreamPartitions(ctx context.Context, feedName string) (int64, error)

ScheduleChangeStreamPartitions checks each partition in created state, and if all its parent partitions are finished, it will update its state to scheduled.

func (*SpannerAdapter) SetObjectExactVersionLegalHold added in v1.113.1 

func (s *SpannerAdapter) SetObjectExactVersionLegalHold(ctx context.Context, opts SetObjectExactVersionLegalHold) (err error)

SetObjectExactVersionLegalHold sets the legal hold configuration of an exact version of an object.

func (*SpannerAdapter) SetObjectExactVersionRetention added in v1.110.1 

func (s *SpannerAdapter) SetObjectExactVersionRetention(ctx context.Context, opts SetObjectExactVersionRetention) (err error)

SetObjectExactVersionRetention sets the retention configuration of an exact version of an object.

func (*SpannerAdapter) SetObjectLastCommittedLegalHold added in v1.113.1 

func (s *SpannerAdapter) SetObjectLastCommittedLegalHold(ctx context.Context, opts SetObjectLastCommittedLegalHold) (err error)

SetObjectLastCommittedLegalHold sets the legal hold configuration of the most recently committed version of an object.

func (*SpannerAdapter) SetObjectLastCommittedRetention added in v1.110.1 

func (s *SpannerAdapter) SetObjectLastCommittedRetention(ctx context.Context, opts SetObjectLastCommittedRetention) (err error)

SetObjectLastCommittedRetention sets the retention configuration of the most recently committed version of an object.

func (*SpannerAdapter) SpannerMigration added in v1.114.2 

func (s *SpannerAdapter) SpannerMigration() *migrate.Migration

SpannerMigration returns steps needed for migrating spanner database.

func (*SpannerAdapter) TestCreateChangeStream added in v1.138.2 

func (s *SpannerAdapter) TestCreateChangeStream(ctx context.Context, name string) error

TestCreateChangeStream creates a change stream for testing purposes.

func (*SpannerAdapter) TestCreateChangeStreamMetadata added in v1.144.3 

func (s *SpannerAdapter) TestCreateChangeStreamMetadata(ctx context.Context, name string) error

TestCreateChangeStreamMetadata creates only the metadata table and index for testing purposes.

func (*SpannerAdapter) TestDeleteChangeStream added in v1.138.2 

func (s *SpannerAdapter) TestDeleteChangeStream(ctx context.Context, name string) error

TestDeleteChangeStream deletes the change stream with the given name for testing purposes.

func (*SpannerAdapter) TestDeleteChangeStreamMetadata added in v1.144.3 

func (s *SpannerAdapter) TestDeleteChangeStreamMetadata(ctx context.Context, name string) error

TestDeleteChangeStreamMetadata deletes only the metadata table and index for testing purposes.

func (*SpannerAdapter) TestMigrateToLatest added in v1.110.1 

func (s *SpannerAdapter) TestMigrateToLatest(ctx context.Context) error

TestMigrateToLatest creates a database and applies all the migration for test purposes.

func (*SpannerAdapter) TestingBatchInsertObjects added in v1.106.1 

func (s *SpannerAdapter) TestingBatchInsertObjects(ctx context.Context, objects []RawObject) (err error)

TestingBatchInsertObjects batch inserts objects for testing.

func (*SpannerAdapter) TestingBatchInsertSegments added in v1.103.2 

func (s *SpannerAdapter) TestingBatchInsertSegments(ctx context.Context, aliasCache *NodeAliasCache, segments []RawSegment) (err error)

TestingBatchInsertSegments implements SpannerAdapter.

func (*SpannerAdapter) TestingDeleteAll added in v1.103.2 

func (s *SpannerAdapter) TestingDeleteAll(ctx context.Context) (err error)

TestingDeleteAll implements Adapter.

func (*SpannerAdapter) TestingGetAllObjects added in v1.104.1 

func (s *SpannerAdapter) TestingGetAllObjects(ctx context.Context) (_ []RawObject, err error)

TestingGetAllObjects returns the state of the database.

func (*SpannerAdapter) TestingGetAllSegments added in v1.103.2 

func (s *SpannerAdapter) TestingGetAllSegments(ctx context.Context, aliasCache *NodeAliasCache) (segments []RawSegment, err error)

TestingGetAllSegments implements Adapter.

func (*SpannerAdapter) TestingSetObjectCreatedAt added in v1.147.4 

func (s *SpannerAdapter) TestingSetObjectCreatedAt(ctx context.Context, object ObjectStream, createdAt time.Time) (rowsAffected int64, err error)

TestingSetObjectCreatedAt sets the created_at of the object to the given value in tests.

func (*SpannerAdapter) TestingSetObjectVersion added in v1.112.2 

func (s *SpannerAdapter) TestingSetObjectVersion(ctx context.Context, object ObjectStream, randomVersion Version) (rowsAffected int64, err error)

TestingSetObjectVersion sets the version of the object to the given value.

func (*SpannerAdapter) TestingSetPlacementAllSegments added in v1.112.2 

func (s *SpannerAdapter) TestingSetPlacementAllSegments(ctx context.Context, placement storj.PlacementConstraint) (err error)

TestingSetPlacementAllSegments sets the placement of all segments to the given value.

func (*SpannerAdapter) UncoordinatedDeleteAllBucketObjects added in v1.132.2 

func (s *SpannerAdapter) UncoordinatedDeleteAllBucketObjects(ctx context.Context, opts UncoordinatedDeleteAllBucketObjects) (totalDeletedObjects, totalDeletedSegments int64, err error)

UncoordinatedDeleteAllBucketObjects deletes objects in the specified bucket in batches of opts.BatchSize number of objects.

func (*SpannerAdapter) UnderlyingDB added in v1.107.1 

func (s *SpannerAdapter) UnderlyingDB() *recordeddb.SpannerClient

UnderlyingDB returns a handle to the underlying DB.

func (*SpannerAdapter) UpdateChangeStreamPartitionState added in v1.141.5 

func (s *SpannerAdapter) UpdateChangeStreamPartitionState(ctx context.Context, feedName, partitionToken string, newState changestream.PartitionState) error

UpdateChangeStreamPartitionState updates the watermark for a change stream partition in the metabase.

func (*SpannerAdapter) UpdateChangeStreamPartitionWatermark added in v1.141.5 

func (s *SpannerAdapter) UpdateChangeStreamPartitionWatermark(ctx context.Context, feedName, partitionToken string, newWatermark time.Time) error

UpdateChangeStreamPartitionWatermark updates the watermark for a change stream partition in the metabase.

func (*SpannerAdapter) UpdateObjectLastCommittedMetadata added in v1.106.1 

func (s *SpannerAdapter) UpdateObjectLastCommittedMetadata(ctx context.Context, opts UpdateObjectLastCommittedMetadata) (affected int64, err error)

UpdateObjectLastCommittedMetadata updates an object metadata.

func (*SpannerAdapter) UpdateSegmentPieces added in v1.106.1 

func (s *SpannerAdapter) UpdateSegmentPieces(ctx context.Context, opts UpdateSegmentPieces, oldPieces, newPieces AliasPieces) (resultPieces AliasPieces, err error)

UpdateSegmentPieces updates pieces for specified segment, if pieces matches oldPieces.

func (*SpannerAdapter) UpdateTableStats added in v1.107.1 

func (s *SpannerAdapter) UpdateTableStats(ctx context.Context) error

UpdateTableStats forces an update of table statistics. Probably useful mostly in test scenarios.

func (*SpannerAdapter) WithTx added in v1.104.1 

func (s *SpannerAdapter) WithTx(ctx context.Context, opts TransactionOptions, f func(context.Context, TransactionAdapter) error) error

WithTx provides a TransactionAdapter for the context of a database transaction.

type SpannerConfig added in v1.103.2 

type SpannerConfig struct {
	Database        string `help:"Database definition for spanner connection in the form  projects/P/instances/I/databases/DB"`
	ApplicationName string `help:"Application name to be used in spanner client as a tag for queries and transactions"`
	Compression     string `help:"Compression type to be used in spanner client for gRPC calls (gzip)"`

	HealthCheckWorkers  int           `hidden:"true" help:"Number of workers used by health checker for the connection pool." default:"10" testDefault:"1"`
	HealthCheckInterval time.Duration `hidden:"true" help:"How often the health checker pings a session." default:"50m"`
	MinOpenedSesssions  uint64        `hidden:"true" help:"Minimum number of sessions that client tries to keep open." default:"100"`
	TrackSessionHandles bool          `hidden:"true" help:"Track session handles." default:"false" testDefault:"true"`
}

SpannerConfig includes all the configuration required by using spanner.

func NewTestSpannerConfig added in v1.103.2 

func NewTestSpannerConfig(database SpannerTestDatabase) SpannerConfig

NewTestSpannerConfig creates SpannerConfig for testing.

type SpannerTestDatabase added in v1.103.2 

type SpannerTestDatabase struct {
	// contains filtered or unexported fields
}

SpannerTestDatabase manages Spanner database and migration for tests.

func NewSpannerTestDatabase added in v1.104.1 

func NewSpannerTestDatabase(ctx context.Context, logger *zap.Logger, connstr string, withMigration bool) (SpannerTestDatabase, error)

NewSpannerTestDatabase creates the database (=creates / migrates the database).

func (SpannerTestDatabase) Close added in v1.103.2 

func (d SpannerTestDatabase) Close() error

Close drops the temporary test database.

func (SpannerTestDatabase) Connection added in v1.104.1 

func (d SpannerTestDatabase) Connection() string

Connection returns with the used connection string (with added unique suffix).

type StreamIDCursor 

type StreamIDCursor struct {
	StreamID uuid.UUID
}

StreamIDCursor is a cursor used during iteration through streamIDs of a pending object.

type StreamIDSuffix added in v1.123.4 

type StreamIDSuffix [8]byte

StreamIDSuffix is the last 8 bytes of an object's stream ID. It's used together with an object's internal version ID to produce a version ID for the public API.

func (StreamIDSuffix) EncodeSpanner added in v1.123.4 

func (s StreamIDSuffix) EncodeSpanner() (any, error)

EncodeSpanner implements spanner.Encoder.

func (StreamIDSuffix) IsZero added in v1.123.4 

func (s StreamIDSuffix) IsZero() bool

IsZero returns whether all bytes in the StreamIDSuffix are 0.

func (StreamIDSuffix) Value added in v1.123.4 

func (s StreamIDSuffix) Value() (driver.Value, error)

Value implements sql/driver.Valuer.

type StreamRange 

type StreamRange struct {
	PlainStart int64
	PlainLimit int64 // limit is exclusive
}

StreamRange allows to limit stream positions based on the plain offsets.

type StreamVersionID added in v1.95.1 

type StreamVersionID uuid.UUID

StreamVersionID represents combined Version and StreamID suffix for purposes of public API. First 8 bytes represents Version and rest are object StreamID suffix. TODO(ver): we may consider renaming this type to VersionID but to do that we would need to rename metabase.Version into metabase.SequenceNumber/metabase.Sequence to avoid confusion.

func NewStreamVersionID added in v1.110.1 

func NewStreamVersionID(version Version, streamID uuid.UUID) StreamVersionID

NewStreamVersionID returns a new stream version id.

func StreamVersionIDFromBytes added in v1.95.1 

func StreamVersionIDFromBytes(bytes []byte) (_ StreamVersionID, err error)

StreamVersionIDFromBytes decodes stream version id from bytes.

func (StreamVersionID) Bytes added in v1.95.1 

func (s StreamVersionID) Bytes() []byte

Bytes returnes stream version id bytes.

func (StreamVersionID) IsZero added in v1.121.2 

func (s StreamVersionID) IsZero() bool

IsZero returns whether all bytes in the StreamVersionID are 0.

func (*StreamVersionID) SetStreamID added in v1.121.2 

func (s *StreamVersionID) SetStreamID(streamID uuid.UUID)

SetStreamID encodes the provided stream ID into the stream version ID.

func (StreamVersionID) StreamIDSuffix added in v1.95.1 

func (s StreamVersionID) StreamIDSuffix() StreamIDSuffix

StreamIDSuffix returns StreamID suffix encoded into stream version id.

func (StreamVersionID) Version added in v1.95.1 

func (s StreamVersionID) Version() Version

Version returns Version encoded into stream version id.

type TableStats added in v1.32.2 

type TableStats struct {
	SegmentCount int64
}

TableStats contains information about the metabase status.

type TransactionAdapter added in v1.104.1 

type TransactionAdapter interface {
	// contains filtered or unexported methods
}

TransactionAdapter is a low level extension point to use datasource related queries inside of a transaction.

type TransactionOptions added in v1.126.5 

type TransactionOptions struct {
	// supported only by Spanner.
	MaxCommitDelay *time.Duration
	TransactionTag string

	// supported only by Spanner.
	TransmitEvent bool
}

TransactionOptions contains options for transaction.

type UncoordinatedDeleteAllBucketObjects added in v1.138.2 

type UncoordinatedDeleteAllBucketObjects struct {
	Bucket    BucketLocation
	BatchSize int

	// supported only by Spanner.
	StalenessTimestampBound spanner.TimestampBound
	MaxCommitDelay          *time.Duration

	// OnObjectsDeleted is called per batch with object info for deleted objects in that batch.
	// When nil, object info is not collected.
	OnObjectsDeleted func([]DeleteObjectsInfo)
}

UncoordinatedDeleteAllBucketObjects contains arguments for deleting a whole bucket.

type UpdateObjectLastCommittedMetadata added in v1.91.2 

type UpdateObjectLastCommittedMetadata struct {
	ObjectLocation
	StreamID uuid.UUID

	EncryptedUserData
	// SetEncryptedETag is true for new uplink clients that know to send EncryptedETag.
	SetEncryptedETag bool
}

UpdateObjectLastCommittedMetadata contains arguments necessary for replacing an object metadata.

func (*UpdateObjectLastCommittedMetadata) Verify added in v1.91.2 

func (obj *UpdateObjectLastCommittedMetadata) Verify() error

Verify object stream fields.

type UpdateSegmentPieces 

type UpdateSegmentPieces struct {
	// Name of the database adapter to use for this segment. If empty (""), check all adapters
	// until the segment is found.
	DBAdapterName string

	StreamID uuid.UUID
	Position SegmentPosition

	OldPieces Pieces

	NewRedundancy storj.RedundancyScheme
	NewPieces     Pieces

	NewRepairedAt time.Time // sets new time of last segment repair (optional).
}

UpdateSegmentPieces contains arguments necessary for updating segment pieces.

type VerifySegment added in v1.64.1 

type VerifySegment struct {
	StreamID uuid.UUID
	Position SegmentPosition

	CreatedAt  time.Time
	RepairedAt *time.Time

	RootPieceID storj.PieceID
	Redundancy  storj.RedundancyScheme

	AliasPieces AliasPieces
}

VerifySegment result of listing segments for verifying remote segments.

type Version 

type Version int64

Version is used to uniquely identify objects with the same key.

func (*Version) DecodeSpanner added in v1.143.2 

func (v *Version) DecodeSpanner(val any) error

DecodeSpanner implements spanner.Decoder.

func (Version) EncodeSpanner added in v1.143.2 

func (v Version) EncodeSpanner() (any, error)

EncodeSpanner implements spanner.Encoder.

func (*Version) Scan added in v1.128.3 

func (v *Version) Scan(val any) error

Scan implements sql.Scanner interface.

func (Version) Value added in v1.143.2 

func (v Version) Value() (driver.Value, error)

Value converts a Version to a database field.

Source Files

View all Source files

Directories

Path Synopsis
Package avrometabase provides utilities for parsing segment metadata from Avro files, enabling bulk import and processing of segment data from external sources.
 Package avrometabase provides utilities for parsing segment metadata from Avro files, enabling bulk import and processing of segment data from external sources.
Package changestream provides Spanner-specific change data capture (CDC) functionality for real-time processing of metabase data changes.
 Package changestream provides Spanner-specific change data capture (CDC) functionality for real-time processing of metabase data changes.
Package metabasetest provides testing utilities and helpers for metabase operations, enabling declarative test steps and simplified test data creation.
 Package metabasetest provides testing utilities and helpers for metabase operations, enabling declarative test steps and simplified test data creation.
Package rangedloop provides a parallel processing framework for iterating over all segments in the metabase, enabling efficient large-scale background operations.
 Package rangedloop provides a parallel processing framework for iterating over all segments in the metabase, enabling efficient large-scale background operations.
rangedlooptest
Package rangedlooptest provides testing utilities for the ranged loop package, including mock providers, simple observers, and helpers for testing ranged loop functionality.
 Package rangedlooptest provides testing utilities for the ranged loop package, including mock providers, simple observers, and helpers for testing ranged loop functionality.
Package zombiedeletion provides automatic cleanup of "zombie" objects - pending objects from failed or abandoned uploads that were never committed or explicitly deleted.
 Package zombiedeletion provides automatic cleanup of "zombie" objects - pending objects from failed or abandoned uploads that were never committed or explicitly deleted.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.