Documentation
¶
Index ¶
- func Dereference[TValue any, TReference any](ctx context.Context, reader DecodingObjectReader[TReference, TValue], ...) (TValue, error)
- func MaybeDereference[TValue any, TReference any](ctx context.Context, reader DecodingObjectReader[TReference, TValue], ...) (TValue, error)
- type DecodingObjectReader
- func LookupParsedObjectReader[TReference object.BasicReference, TParsedObject any](ingester *ParsedObjectPoolIngester[TReference], ...) DecodingObjectReader[TReference, TParsedObject]
- func NewParsedObjectReader[TReference, TParsedObject any](rawReader ObjectReader[TReference, model_core.Message[[]byte, TReference]], ...) DecodingObjectReader[TReference, TParsedObject]
- type MessageObjectParser
- func NewEncodedObjectParser[TReference any](decoder model_encoding.BinaryDecoder) MessageObjectParser[TReference, []byte]
- func NewProtoListObjectParser[TReference any, TMessage any, TMessagePtr interface{ ... }]() MessageObjectParser[TReference, []TMessagePtr]
- func NewProtoObjectParser[TReference any, TMessage any, TMessagePtr interface{ ... }]() MessageObjectParser[TReference, TMessagePtr]
- type MessageObjectReader
- type ObjectParser
- type ObjectReader
- type ParsedObjectEvictionKey
- type ParsedObjectPool
- type ParsedObjectPoolIngester
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Dereference ¶
func Dereference[ TValue any, TReference any, ]( ctx context.Context, reader DecodingObjectReader[TReference, TValue], m model_core.Message[*model_core_pb.DecodableReference, TReference], ) (TValue, error)
Dereference a reference message, returning the value that's associated with it.
func MaybeDereference ¶
func MaybeDereference[ TValue any, TReference any, ]( ctx context.Context, reader DecodingObjectReader[TReference, TValue], m model_core.Message[*model_core_pb.DecodableReference, TReference], ) (TValue, error)
MaybeDereference is identical to Dereference, except that it returns a default instance in case the reference message is not set.
Types ¶
type DecodingObjectReader ¶
type DecodingObjectReader[TReference, TParsedObject any] interface { ObjectReader[model_core.Decodable[TReference], TParsedObject] GetDecodingParametersSizeBytes() int }
DecodingObjectReader is an ObjectReader that assumes that the references that are provided contain decoding parameters (e.g., parameters needed to decrypt the object). Because of this, the implementation is going to be backed by a BinaryDecoder, which permits querying the size of the decoding parameters that are accepted.
func LookupParsedObjectReader ¶
func LookupParsedObjectReader[TReference object.BasicReference, TParsedObject any]( ingester *ParsedObjectPoolIngester[TReference], parser ObjectParser[TReference, TParsedObject], ) DecodingObjectReader[TReference, TParsedObject]
LookupParsedObjectReader returns an object reader that can be used to read parsed contents of an object (e.g., unmarshaled Protobuf messages), given a parser. If a pool is provided, the parsed object contents will be stored in the pool, so that subsequent attempts to access the objects return the previously parsed contents.
func NewParsedObjectReader ¶
func NewParsedObjectReader[TReference, TParsedObject any](rawReader ObjectReader[TReference, model_core.Message[[]byte, TReference]], parser ObjectParser[TReference, TParsedObject]) DecodingObjectReader[TReference, TParsedObject]
NewParsedObjectReader creates a decorator for ObjectReader that parses objects after they have been downloaded.
type MessageObjectParser ¶
type MessageObjectParser[TReference, TMessage any] = ObjectParser[ TReference, model_core.Message[TMessage, TReference], ]
MessageObjectParser is an ObjectParser that returns the parsed object in the form of a model_core.Message. This applies most of the basic ObjectParser implementations, like the ones that only parse Protobuf messages. However, more complex ones like the one that parses file contents lists may completely convert the message to a native type.
func NewEncodedObjectParser ¶
func NewEncodedObjectParser[ TReference any, ](decoder model_encoding.BinaryDecoder) MessageObjectParser[TReference, []byte]
NewEncodedObjectParser creates an ObjectParser that decodes objects. Decoding operations may include decompression and decryption.
func NewProtoListObjectParser ¶
func NewProtoListObjectParser[ TReference any, TMessage any, TMessagePtr interface { *TMessage proto.Message }, ]() MessageObjectParser[TReference, []TMessagePtr]
NewProtoListObjectParser creates an ObjectParser that is capable of unmarshaling objects containing lists of Protobuf messages. Messages are prefixed with their size, encoded as a variable length integer.
func NewProtoObjectParser ¶
func NewProtoObjectParser[ TReference any, TMessage any, TMessagePtr interface { *TMessage proto.Message }, ]() MessageObjectParser[TReference, TMessagePtr]
NewProtoObjectParser creates an ObjectParser that is capable of unmarshaling objects containing Protobuf messages.
type MessageObjectReader ¶
type MessageObjectReader[TReference, TMessage any] = DecodingObjectReader[ TReference, model_core.Message[TMessage, TReference], ]
MessageObjectReader can be used to read the contents of an object from storage and gain access to its parsed contents. This is a high level convenience type on top of ObjectReader, as a couple of assumptions have been made:
The reference of the object to be read has decoding parameters, which are necessary to successfully decode the object. Decoding parameters may include a tag/nonce that is needed to decrypt the object's contents.
The object's contents are returned using a Message, meaning references to any child objects are returned as well.
The reference type of the object and that of its children are identical. For high level code, this tends to be a common assumption, as it's necessary for easy graph traversal.
type ObjectParser ¶
type ObjectParser[TReference, TParsedObject any] interface { ParseObject(in model_core.Message[[]byte, TReference], decodingParameters []byte) (TParsedObject, error) AppendUniqueKeys(keys []unique.Handle[any]) []unique.Handle[any] GetDecodingParametersSizeBytes() int }
ObjectParser is used by ParsingObjectReader to parse objects after they have been read from storage. Parsing steps may include decoding (decompression/decryption), but also unmarshaling.
The same object may be parseable by multiple ObjectParsers. For example, for some objects it may be preferential to first scan them and create an index, so that subsequent access is faster. This might result in multiple ObjectParser implementations, each of which is used in different circumstances.
Types like ParsedObjectPool require being able to uniquely identify instances of ObjectParser, so that the key used to identify objects in storage does not collide. The AppendUniqueKeys() method returns a sequence of unique handles.
func NewChainedObjectParser ¶
func NewChainedObjectParser[TReference, TParsedObject any](parserA MessageObjectParser[TReference, []byte], parserB ObjectParser[TReference, TParsedObject]) ObjectParser[TReference, TParsedObject]
NewChainedObjectParser chains two ObjectParsers together, returning a single instance that parses objects by running them through both underlying implementations.
An example use case for this is to parse objects containing compressed/encrypted Protobuf messages. Such objects first need to be decompressed/decrypted before getting unmarshaled.
func NewRawObjectParser ¶
func NewRawObjectParser[TReference any]() ObjectParser[TReference, []byte]
NewRawObjectParser creates an ObjectParser that assumes that the object is a leaf that contains binary data, such as a chunk of a file.
type ObjectReader ¶
type ObjectReader[TReference, TParsedObject any] interface { ReadObject(ctx context.Context, reference TReference) (TParsedObject, error) }
ObjectReader can be used to read an object from storage and return its contents. This is a generalization of object.Downloader, in that contents can also be returned in decoded and parsed form.
func NewDownloadingObjectReader ¶
func NewDownloadingObjectReader[TReference any](downloader object.Downloader[TReference]) ObjectReader[TReference, model_core.Message[[]byte, object.LocalReference]]
NewDownloadingObjectReader creates a ObjectReader that reads objects by downloading them from storage. Storage can either be local (e.g., backed by a disk) or remote (e.g., via gRPC).
type ParsedObjectEvictionKey ¶
type ParsedObjectEvictionKey struct {
// contains filtered or unexported fields
}
ParsedObjectEvictionKey is the key that ParsedObjectPool uses to identify cached objects. It needs to be used as the value type of the eviction set that is used to remove cached objects if the maximum size of the pool has been reached.
type ParsedObjectPool ¶
type ParsedObjectPool struct {
// contains filtered or unexported fields
}
ParsedObjectPool is a pool of recently accessed objects, for which the parsed contents are stored. This allows repeated access to the same object to skip downloading and parsing.
func NewParsedObjectPool ¶
func NewParsedObjectPool(evictionSet eviction.Set[ParsedObjectEvictionKey], maximumCount, maximumSizeBytes int) *ParsedObjectPool
NewParsedObjectPool creates a ParsedObjectPool that is initially empty.
func NewParsedObjectPoolFromConfiguration ¶
func NewParsedObjectPoolFromConfiguration(configuration *model_parser_cfg_pb.ParsedObjectPool) (*ParsedObjectPool, error)
NewParsedObjectPoolFromConfiguration creates a ParsedObjectPool whose size and eviction policy correspond to parameters provided in a configuration file.
type ParsedObjectPoolIngester ¶
type ParsedObjectPoolIngester[TReference any] struct { // contains filtered or unexported fields }
ParsedObjectPoolIngester associates a ParsedObjectPool with an ObjectReader that is responsible for reading raw object contents from storage.
func NewParsedObjectPoolIngester ¶
func NewParsedObjectPoolIngester[TReference any]( pool *ParsedObjectPool, rawReader ObjectReader[TReference, model_core.Message[[]byte, TReference]], ) *ParsedObjectPoolIngester[TReference]
NewParsedObjectPoolIngester creates a ParsedObjectPoolIngester, thereby associating ParsedObjectPool with a ParsedObjectReader for reading raw object contents from storage.
It is permitted to set the provided pool to nil. In that case LookupParsedObjectReader() will create readers that do not perform any caching at all. Any attempt to read a parsed object will lead to the object's contents to be read from storage.
Source Files
¶
- chained_object_parser.go
- configuration.go
- decoding_object_reader.go
- dereference.go
- downloading_object_reader.go
- encoded_object_parser.go
- message_object_parser.go
- message_object_reader.go
- object_parser.go
- object_reader.go
- parsed_object_pool.go
- parsed_object_reader.go
- proto_list_object_parser.go
- proto_object_parser.go
- raw_object_parser.go