nodeselection

package
v1.148.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 2, 2026 License: AGPL-3.0 Imports: 30 Imported by: 7

Documentation

Overview

Package uploadselection implements node selection logic for uploads.

Package nodeselection provides functionality for selecting storage nodes.

Index

Constants

View Source 
const (
	// AutoExcludeSubnet is placement annotation key to turn off subnet
	// restrictions.
	AutoExcludeSubnet = "autoExcludeSubnet"

	// AutoExcludeSubnetOFF is the value of AutoExcludeSubnet to disable subnet
	// restrictions.
	AutoExcludeSubnetOFF = "off"

	// Location is the placement annotation key for meaningful and
	// human-readable descriptions of placements.
	Location = "location"
)

Variables

View Source 
var EeaCountriesWithoutEu = []location.CountryCode{
	location.Iceland,
	location.Liechtenstein,
	location.Norway,
}

EeaCountriesWithoutEu defined the EEA countries.

View Source 
var ErrNotEnoughNodes = errs.Class("not enough nodes")

ErrNotEnoughNodes is when selecting nodes failed with the given parameters.

View Source 
var ErrPlacement = errs.Class("placement")

ErrPlacement is used for placement definition related parsing errors.

View Source 
var (

	// Error represents an uploadselection error.
	Error = errs.Class("uploadselection")
)
View Source 
var EuCountries = []location.CountryCode{
	location.Austria,
	location.Belgium,
	location.Bulgaria,
	location.Croatia,
	location.Cyprus,
	location.Czechia,
	location.Denmark,
	location.Estonia,
	location.Finland,
	location.France,
	location.Germany,
	location.Greece,
	location.Hungary,
	location.Ireland,
	location.Italy,
	location.Lithuania,
	location.Latvia,
	location.Luxembourg,
	location.Malta,
	location.TheNetherlands,
	location.Poland,
	location.Portugal,
	location.Romania,
	location.Slovenia,
	location.Slovakia,
	location.Spain,
	location.Sweden,
}

EuCountries defines the member countries of European Union.

View Source 
var LastNetAttribute = mustCreateNodeAttribute("last_net")

LastNetAttribute is used for subnet based declumping/selection.

Functions

func AddArithmetic added in v1.124.4 

func AddArithmetic(in map[any]interface{}) map[any]interface{}

AddArithmetic defines generic math operation for various types.

func ChoiceOfNStream added in v1.126.4 

func ChoiceOfNStream(n int64, score ScoreNode) func(NodeStream) NodeStream

ChoiceOfNStream creates a stream processor that selects the best node from each batch of n nodes. The best node is determined by the highest score according to the provided ScoreNode.

func ConvertType added in v1.124.4 

func ConvertType(b any, t reflect.Type) (any, error)

ConvertType tries to convert a type for the most generic mathmetical type which supports math operations.

func DailyPeriods added in v1.134.1 

func DailyPeriods(values ...int64) func() int64

DailyPeriods returns a function, which returns the value of the period in days based on the current hour.

func DailyPeriodsForHour added in v1.134.1 

func DailyPeriodsForHour(hour int, values []int64) int64

DailyPeriodsForHour returns the value of the period in days based on the given hour.

func EqualSelector added in v1.107.1 

func EqualSelector(nodeAttribute NodeAttribute, attribute string) func(SelectedNode) bool

EqualSelector returns a function that compares the node attribute with the given attribute.

func GetAnnotation added in v1.86.1 

func GetAnnotation(filter NodeFilter, name string) string

GetAnnotation retrieves annotation from AnnotatedNodeFilter.

func GroupConstraint added in v1.126.4 

func GroupConstraint(attribute NodeAttribute, max int64) func([]*SelectedNode, *SelectedNode) bool

GroupConstraint creates a constraint that limits the number of nodes with the same attribute value.

func Module added in v1.120.1 

func Module(ball *mud.Ball)

Module is a mud module.

func NodeAttributes added in v1.126.2 

func NodeAttributes(attributes []NodeAttribute, separator string) func(node SelectedNode) string

NodeAttributes is a collection of multiple NodeAttribute.

func RoundWithProbability added in v1.108.1 

func RoundWithProbability(r float64) int

RoundWithProbability is like math.Round, but instead of rounding 2.6 to 3 all the time, it will round up to 3 with 60% chance, and to 2 with 40% chance.

func StreamFilter added in v1.126.4 

func StreamFilter(filter StreamConstraint) func(stream NodeStream) NodeStream

StreamFilter creates a Node selector based on streaming constructs. Streaming can select unbounded number of nodes until it finds enough good (or no more nodes). Can be slow.

Types

type AllowedNodesFilter added in v1.97.1 

type AllowedNodesFilter []storj.NodeID

AllowedNodesFilter is a special filter which enables only the selected nodes.

func AllowedNodesFromFile added in v1.97.1 

func AllowedNodesFromFile(file string) (AllowedNodesFilter, error)

AllowedNodesFromFile loads a list of allowed NodeIDs from a text file. One ID per line.

func (AllowedNodesFilter) Match added in v1.97.1 

func (n AllowedNodesFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

type AnnotatedNodeFilter added in v1.86.1 

type AnnotatedNodeFilter struct {
	Filter      NodeFilter
	Annotations []Annotation
}

AnnotatedNodeFilter is just a NodeFilter with additional annotations.

func (AnnotatedNodeFilter) GetAnnotation added in v1.87.1 

func (a AnnotatedNodeFilter) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (AnnotatedNodeFilter) Match added in v1.88.2 

func (a AnnotatedNodeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

func (AnnotatedNodeFilter) String added in v1.89.2 

func (a AnnotatedNodeFilter) String() string

type Annotation added in v1.87.1 

type Annotation struct {
	Key   string
	Value string
}

Annotation can be used as node filters in 'XX && annotation('...')' like struct.

func (Annotation) GetAnnotation added in v1.87.1 

func (a Annotation) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (Annotation) Match added in v1.88.2 

func (a Annotation) Match(node *SelectedNode) bool

Match implements NodeFilter.

func (Annotation) String added in v1.89.2 

func (a Annotation) String() string

type AnyFilter added in v1.86.1 

type AnyFilter struct{}

AnyFilter matches all the nodes.

func (AnyFilter) Match added in v1.88.2 

func (a AnyFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type AttributeFilter added in v1.100.2 

type AttributeFilter struct {
	// contains filtered or unexported fields
}

AttributeFilter selects nodes based on dynamic node attributes (eg. vetted=true or tag:owner=...).

func NewAttributeFilter added in v1.100.2 

func NewAttributeFilter(attr string, testStr string, value any) (*AttributeFilter, error)

NewAttributeFilter creates new attribute filter. testStr is the type of equality test to perform, can be "=", "==", "!=", "<>", "<", "<=", ">", ">=". If value is stringNotMatch, then the test is inverted.

func (*AttributeFilter) Match added in v1.100.2 

func (a *AttributeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

type CohortName added in v1.139.1 

type CohortName func(n SelectedNode) []byte

CohortName will generate a name for a specific cohort. For example, let's say that we want the Uplink to consider unique racks across data centers as individual cohorts? We want all nodes that share the same DC and Rack to have the same cohort name. Perhaps we have DCs ewr1 and fra2, and in each one, racks 1, 2, and 3. So there is a node tag of "dc" with values "ewr1" and another node tag of "rack" with value "1". So what a cohort name might be is "ewr1-1" or "fra2-3". This is configured in the cohort requirements definition as `tag($SIGNER_ZERO, "dc") + "-" + tag($SIGNER_ZERO, "rack")`.

type CohortRequirements added in v1.139.1 

type CohortRequirements pb.CohortRequirements

CohortRequirements is a configured set of cohort requirements for uploads.

func CohortRequirementsFromString added in v1.139.1 

func CohortRequirementsFromString(val string) (*CohortRequirements, map[string]CohortName, error)

CohortRequirementsFromString parses a cohort requirement definition.

func (*CohortRequirements) ToProto added in v1.139.1 

func (c *CohortRequirements) ToProto() *pb.CohortRequirements

ToProto converts the parsed CohortRequirements to a protobuf.

type CompareNodes added in v1.128.6 

type CompareNodes func(uplink storj.NodeID) func(node1 *SelectedNode, node2 *SelectedNode) int

CompareNodes compare two nodes for a specific client (uplink). Returns < 0 if node2 is better, returns > 0 if node1 is better, and returns 0 if both are equal.

func Compare added in v1.128.6 

func Compare(scoreNodes ...ScoreNode) CompareNodes

Compare creates a CompareNodes function from multiple ScoreNode parameters. It compares nodes by evaluating each score in order and using the first non-equal result. If all scores are equal, it returns 0.

type ConfigurablePlacementRule added in v1.95.1 

type ConfigurablePlacementRule struct {
	PlacementRules string
}

ConfigurablePlacementRule is a string configuration includes all placement rules in the form of id1:def1,id2:def2...

func (ConfigurablePlacementRule) Parse added in v1.95.1 

func (c ConfigurablePlacementRule) Parse(defaultPlacement func() (Placement, error), environment PlacementConfigEnvironment) (PlacementDefinitions, error)

Parse creates the PlacementDefinitions from the string rules. defaultPlacement is used to create the placement if no placement has been set.

func (*ConfigurablePlacementRule) Set added in v1.95.1 

func (c *ConfigurablePlacementRule) Set(s string) error

Set implements pflag.Value.

func (*ConfigurablePlacementRule) String added in v1.95.1 

func (c *ConfigurablePlacementRule) String() string

String implements pflag.Value.

func (*ConfigurablePlacementRule) Type added in v1.95.1 

func (c *ConfigurablePlacementRule) Type() string

Type implements pflag.Value.

type CountryFilter added in v1.84.1 

type CountryFilter struct {
	// contains filtered or unexported fields
}

CountryFilter can select nodes based on the condition of the country code.

func NewContinentFilterFromString added in v1.112.2 

func NewContinentFilterFromString(continent string) (*CountryFilter, error)

NewContinentFilterFromString parses country definitions like 'SA','!NA'.

func NewCountryFilter added in v1.84.1 

func NewCountryFilter(permit location.Set) *CountryFilter

NewCountryFilter creates a new CountryFilter.

func NewCountryFilterFromString added in v1.89.2 

func NewCountryFilterFromString(countries []string) (*CountryFilter, error)

NewCountryFilterFromString parses country definitions like 'hu','!hu','*','none' and creates a CountryFilter.

func (*CountryFilter) Match added in v1.88.2 

func (p *CountryFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (*CountryFilter) String added in v1.89.2 

func (p *CountryFilter) String() string

type DownloadCase added in v1.139.3 

type DownloadCase struct {
	// contains filtered or unexported fields
}

DownloadCase represents a conditional selector case for DownloadSwitch.

func NewDownloadCase added in v1.139.3 

func NewDownloadCase(condition DownloadCondition, selector DownloadSelector) DownloadCase

NewDownloadCase creates a new DownloadCase with the given condition and selector.

type DownloadCondition added in v1.139.3 

type DownloadCondition func(ctx context.Context, requestor storj.NodeID) bool

DownloadCondition is a function that determines if a condition is met for a requester.

func RequesterIs added in v1.139.3 

func RequesterIs(targets ...string) DownloadCondition

RequesterIs creates a DownloadCondition that matches if the requester is one of the target nodes.

type DownloadSelector added in v1.109.1 

type DownloadSelector func(ctx context.Context, requester storj.NodeID, possibleNodes map[storj.NodeID]*SelectedNode, needed int) (map[storj.NodeID]*SelectedNode, error)

DownloadSelector will take a map of possible nodes to choose for a download. It returns a new map of nodes to consider for selecting for the download. It is always true that 0 <= len(result) <= len(possibleNodes), and every element in result will have come from possibleNodes. 'needed' is a hint to the selector of how many nodes are needed for return ideally, so many selectors will try to return at least 'needed' nodes. 

var DefaultDownloadSelector DownloadSelector = defaultDownloadSelector

DefaultDownloadSelector is a DownloadSelector that returns the set of possibleNodes unchanged. 

var ExcludeAllDownloadSelector DownloadSelector = excludeAllDownloadSelector

ExcludeAllDownloadSelector is a DownloadSelector that always returns an empty map.

func DownloadBest added in v1.109.1 

func DownloadBest(tracker UploadSuccessTracker) DownloadSelector

DownloadBest will take a set of nodes and will return just the best nodes.

func DownloadChoiceOfN added in v1.109.1 

func DownloadChoiceOfN(comparison CompareNodes, n int64) DownloadSelector

DownloadChoiceOfN will take a set of nodes and winnow it down using choice of n. n is an int64 type due to a mito scripting shortcoming but really an int16 should be fine.

func DownloadFilter added in v1.139.3 

func DownloadFilter(filter NodeFilter, selector DownloadSelector) DownloadSelector

DownloadFilter creates a DownloadSelector that applies a filter before using another selector.

func DownloadSelectorFromString added in v1.109.1 

func DownloadSelectorFromString(expr string, environment PlacementConfigEnvironment) (DownloadSelector, error)

DownloadSelectorFromString parses complex node download selection expressions from config lines.

func DownloadSwitch added in v1.139.3 

func DownloadSwitch(dflt DownloadSelector, cases ...DownloadCase) DownloadSelector

DownloadSwitch creates a DownloadSelector that tries cases in order, then falls back to default.

type ECParameters added in v1.107.1 

type ECParameters struct {
	Minimum int
	Success func(k int) int
	Total   int
	Repair  func(k int) int
}

ECParameters can be used to override certain part of the RS parameters.

func (*ECParameters) UnmarshalYAML added in v1.133.2 

func (e *ECParameters) UnmarshalYAML(unmarshal func(interface{}) error) (err error)

UnmarshalYAML handles YAML unmarshaling for ECParameters.

type ExcludeAllFilter added in v1.84.1 

type ExcludeAllFilter struct{}

ExcludeAllFilter will never select any node.

func (ExcludeAllFilter) Match added in v1.88.2 

func (ExcludeAllFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludeFilter added in v1.86.1 

type ExcludeFilter struct {
	// contains filtered or unexported fields
}

ExcludeFilter excludes only the matched nodes.

func NewExcludeFilter added in v1.86.1 

func NewExcludeFilter(filter NodeFilter) ExcludeFilter

NewExcludeFilter creates filter, nodes matching the given filter will be excluded.

func (ExcludeFilter) Match added in v1.88.2 

func (e ExcludeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (ExcludeFilter) String added in v1.89.2 

func (e ExcludeFilter) String() string

type ExcludedIDs added in v1.84.1 

type ExcludedIDs []storj.NodeID

ExcludedIDs can blacklist NodeIDs.

func (ExcludedIDs) Match added in v1.88.2 

func (e ExcludedIDs) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludedNetworks added in v1.84.1 

type ExcludedNetworks []string

ExcludedNetworks will exclude nodes with specified networks.

func (ExcludedNetworks) Match added in v1.88.2 

func (e ExcludedNetworks) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludedNodeNetworks added in v1.86.1 

type ExcludedNodeNetworks []*SelectedNode

ExcludedNodeNetworks exclude nodes which has same net as the one of the specified.

func (ExcludedNodeNetworks) Match added in v1.88.2 

func (e ExcludedNodeNetworks) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type Invariant added in v1.96.2 

type Invariant func(pieces metabase.Pieces, nodes []SelectedNode) intset.Set

Invariant checks the current placement, and identifies the pieces which should be moved. Used by repair jobs.

func AllGood added in v1.96.2 

func AllGood() Invariant

AllGood is an invariant, which accepts all piece sets as good.

func ClumpingByAnyTag added in v1.96.2 

func ClumpingByAnyTag(key string, maxAllowed int) Invariant

ClumpingByAnyTag tries to limit the number of nodes with the same tag value.

func ClumpingByAttribute added in v1.96.2 

func ClumpingByAttribute(attr NodeAttribute, maxAllowed int) Invariant

ClumpingByAttribute allows only one selected piece by attribute groups.

func CombinedInvariant added in v1.139.8 

func CombinedInvariant(invariants ...Invariant) Invariant

CombinedInvariant combines multiple invariants into one, by taking the union of all sets of bad pieces.

func FilterInvariant added in v1.139.8 

func FilterInvariant(filter NodeFilter) Invariant

FilterInvariant enables marking pieces as OOP (out of placement) based on a filter.

func InvariantFromString added in v1.100.2 

func InvariantFromString(expr string) (Invariant, error)

InvariantFromString parses complex invariants (~declumping rules) from config lines.

type NeedMore added in v1.133.2 

type NeedMore func() func(node *SelectedNode) bool

NeedMore is a stateful function, which returns true, if more nodes are needed.

func AtLeast added in v1.133.2 

func AtLeast(attribute NodeAttribute, min interface{}) NeedMore

AtLeast is a needMore function, which will return true, if the number groups (specified by the given attribute) is less than the minimum value.

type NodeAttribute added in v1.95.1 

type NodeAttribute func(SelectedNode) string

NodeAttribute returns a string (like last_net or tag value) for each SelectedNode. can be used to group / label nodes.

func AnyNodeTagAttribute added in v1.96.2 

func AnyNodeTagAttribute(tagName string) NodeAttribute

AnyNodeTagAttribute selects a tag value from node, accepts any signer.

func CreateNodeAttribute added in v1.95.1 

func CreateNodeAttribute(attr string) (NodeAttribute, error)

CreateNodeAttribute creates the NodeAttribute selected based on a string definition.

func IfSelector added in v1.107.1 

func IfSelector(condition func(SelectedNode) bool, conditionTrue, conditionFalse NodeAttribute) NodeAttribute

IfSelector selects the first node attribute if the condition is true, otherwise the second node attribute.

func NodeTagAttribute added in v1.96.2 

func NodeTagAttribute(signer storj.NodeID, tagName string) NodeAttribute

NodeTagAttribute selects a tag value from node.

func Subnet 

func Subnet(bits int64) NodeAttribute

Subnet can return the IP network of the node for any netmask length.

type NodeFilter added in v1.84.1 

type NodeFilter interface {
	Match(node *SelectedNode) bool
}

NodeFilter can decide if a Node should be part of the selection or not.

func FilterFromString added in v1.100.2 

func FilterFromString(expr string, environment PlacementConfigEnvironment) (NodeFilter, error)

FilterFromString parses complex node filter expressions from config lines.

type NodeFilterFunc added in v1.84.1 

type NodeFilterFunc func(node *SelectedNode) bool

NodeFilterFunc is helper to use func as NodeFilter.

func (NodeFilterFunc) Match added in v1.88.2 

func (n NodeFilterFunc) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type NodeFilterWithAnnotation added in v1.87.1 

type NodeFilterWithAnnotation interface {
	NodeFilter
	GetAnnotation(name string) string
}

NodeFilterWithAnnotation is a NodeFilter with additional annotations.

func WithAnnotation added in v1.86.1 

func WithAnnotation(filter NodeFilter, name string, value string) NodeFilterWithAnnotation

WithAnnotation adds annotations to a NodeFilter.

type NodeFilters added in v1.84.1 

type NodeFilters []NodeFilter

NodeFilters is a collection of multiple node filters (all should vote with true).

func (NodeFilters) GetAnnotation added in v1.87.1 

func (n NodeFilters) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (NodeFilters) Match added in v1.88.2 

func (n NodeFilters) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (NodeFilters) String added in v1.89.2 

func (n NodeFilters) String() string

func (NodeFilters) WithCountryFilter added in v1.84.1 

func (n NodeFilters) WithCountryFilter(permit location.Set) NodeFilters

WithCountryFilter is a helper to create a new filter with additional CountryFilter.

func (NodeFilters) WithExcludedIDs added in v1.84.1 

func (n NodeFilters) WithExcludedIDs(ds []storj.NodeID) NodeFilters

WithExcludedIDs is a helper to create a new filter with additional WithExcludedIDs.

type NodeSelector added in v1.95.1 

type NodeSelector func(ctx context.Context, requester storj.NodeID, n int, excluded []storj.NodeID, alreadySelected []*SelectedNode) ([]*SelectedNode, error)

NodeSelector pick random nodes based on a specific algorithm. Nodes from excluded should never be used. Same is true for alreadySelected, but it may also trigger other restrictions (for example, when a last_net is already selected, all the nodes from the same net should be excluded as well.

type NodeSelectorInit added in v1.95.1 

type NodeSelectorInit func(context.Context, []*SelectedNode, NodeFilter) NodeSelector

NodeSelectorInit initializes a stateful NodeSelector when node cache is refreshed.

func AttributeGroupSelector added in v1.96.2 

func AttributeGroupSelector(attribute NodeAttribute) NodeSelectorInit

AttributeGroupSelector first selects a group with equal chance (like last_net) and choose node from the group randomly.

func BalancedGroupBasedSelector added in v1.97.2 

func BalancedGroupBasedSelector(attribute NodeAttribute, uploadFilter NodeFilter) NodeSelectorInit

BalancedGroupBasedSelector first selects a group with equal chance (like last_net) and choose one single node randomly. . One group can be tried multiple times, and if the node is already selected, it will be ignored.

func BestOfN added in v1.106.1 

func BestOfN(tracker ScoreNode, ratio float64, delegate NodeSelectorInit) NodeSelectorInit

BestOfN selects more nodes than the required one, and choose the fastest from those.

func ChoiceOfN added in v1.106.1 

func ChoiceOfN(comparison CompareNodes, n int64, delegate NodeSelectorInit) NodeSelectorInit

ChoiceOfN will perform the selection for n*x nodes and choose the best node from groups of n size. n is an int64 type due to a mito scripting shortcoming but really an int16 should be fine. NOTE: it may break other pre-conditions, like the results of the balanced selector...

func ChoiceOfNSelection added in v1.113.5 

func ChoiceOfNSelection(n int64, delegate NodeSelectorInit, scoreSource ...ScoreSelection) NodeSelectorInit

ChoiceOfNSelection is similar to ChoiceOfN, but doesn't break the pre-conditions of the original selector. it chooses from selections, without mixing nodes. scoreSources are ar judging the selections in order.

func ChoiceOfTwo added in v1.106.1 

func ChoiceOfTwo(comparer CompareNodes, delegate NodeSelectorInit) NodeSelectorInit

ChoiceOfTwo will repeat the selection and choose the better node pair-wise. NOTE: it may break other pre-conditions, like the results of the balanced selector...

func DualSelector added in v1.108.1 

func DualSelector(fraction float64, first NodeSelectorInit, second NodeSelectorInit) NodeSelectorInit

DualSelector selects fraction of nodes with first, and remaining with the second selector.

func EnoughFast added in v1.108.1 

func EnoughFast(tracker UploadSuccessTracker, ratio float64, splitLine float64, selectionRatio float64, delegate NodeSelectorInit) NodeSelectorInit

EnoughFast will select `ratio` times more nodes. The fastest nodes (under splitLine) will be used the selectionRation nodes, remaining wil be chosen from the second part.

func FilterBest added in v1.106.1 

func FilterBest(tracker UploadSuccessTracker, selection string, uplink string, delegate NodeSelectorInit) NodeSelectorInit

FilterBest is a selector, which keeps only the best nodes (based on percentage, or fixed number of nodes). this selector will permanently ban the worst nodes for the period of nodeselection cache refresh.

func FilterSelector added in v1.97.1 

func FilterSelector(loadTimeFilter NodeFilter, init NodeSelectorInit) NodeSelectorInit

FilterSelector is a specific selector, which can filter out nodes from the upload selection. Note: this is different from the generic filter attribute of the NodeSelectorInit, as that is applied to all node selection (upload/download/repair).

func FilteredSelector added in v1.96.2 

func FilteredSelector(preFilter NodeFilter, init NodeSelectorInit) NodeSelectorInit

FilteredSelector uses another selector on the filtered list of nodes.

func FixedSelector added in v1.137.4 

func FixedSelector(fixed int64, delegate NodeSelectorInit) NodeSelectorInit

FixedSelector selector can override the number of the required nodes, and select more (or less).

func MultiSelector added in v1.137.4 

func MultiSelector(selectors ...NodeSelectorInit) NodeSelectorInit

MultiSelector can combine multiple selectors. It will call each selector in order, and combine the results.

func RandomSelector added in v1.96.2 

func RandomSelector() NodeSelectorInit

RandomSelector selects any nodes with equal chance.

func Reduce added in v1.133.2 

func Reduce(delegate NodeSelectorInit, sortOrder CompareNodes, needMoreChecks ...NeedMore) NodeSelectorInit

Reduce is a NodeSelectorInit, which will reduce the number of nodes selected by the delegate.

func SelectorFromString added in v1.100.2 

func SelectorFromString(expr string, environment PlacementConfigEnvironment) (NodeSelectorInit, error)

SelectorFromString parses complex node selection rules from config lines.

func Stream added in v1.126.4 

func Stream(seed func(nodes []*SelectedNode) NodeStream, steps ...func(NodeStream) NodeStream) NodeSelectorInit

Stream creates a node selector that uses the provided seed function to generate a stream of nodes. Additional processing steps can be applied to the stream before selection.

func TopologySelector added in v1.122.1 

func TopologySelector(weightFunc NodeValue, groups string, selections string, initFilter NodeFilter) NodeSelectorInit

TopologySelector selects nodes using weights and topology structure. Topology is a tree and defined by attributes. (eg. datacenter / server / instance). Number of selected nodes should be defined for each level. It works well, if the number of elements on each level are higher than the requested selection. TODO: existing selection nodes are not used to restrict selection. Repair may use groups too many times.

func UnvettedSelector added in v1.96.2 

func UnvettedSelector(newNodeFraction float64, init NodeSelectorInit) NodeSelectorInit

UnvettedSelector selects new nodes first based on newNodeFraction, and selects old nodes for the remaining.

func WeightedSelector added in v1.116.3 

func WeightedSelector(weightFunc NodeValue, initFilter NodeFilter) NodeSelectorInit

WeightedSelector selects randomly from nodes, but supporting custom probabilities. Each node value is raised to the valuePower power, and then added to valueBallast. Some nodes can be selected more often than others. The implementation is based on Walker's alias method: https://www.youtube.com/watch?v=retAwpUv42E

type NodeSequence added in v1.126.4 

type NodeSequence func(ctx context.Context) *SelectedNode

NodeSequence is a function that returns the next node in a sequence or nil when exhausted.

type NodeStream added in v1.126.4 

type NodeStream func(ctx context.Context, requester storj.NodeID, excluded []storj.NodeID, alreadySelected []*SelectedNode) NodeSequence

NodeStream creates a sequence of nodes, filtering out excluded nodes and those already selected.

func RandomStream added in v1.126.4 

func RandomStream(allNodes []*SelectedNode) NodeStream

RandomStream creates a NodeStream that returns nodes in a random order. It skips nodes that are in the excluded list or already selected.

type NodeTag added in v1.84.1 

type NodeTag struct {
	NodeID   storj.NodeID
	SignedAt time.Time
	Signer   storj.NodeID
	Name     string
	Value    []byte
}

NodeTag is a tag associated with a node (approved by signer).

type NodeTags added in v1.84.1 

type NodeTags []NodeTag

NodeTags is a collection of multiple NodeTag.

func (NodeTags) FindBySignerAndName added in v1.84.1 

func (n NodeTags) FindBySignerAndName(signer storj.NodeID, name string) (NodeTag, error)

FindBySignerAndName selects first tag with same name / NodeID.

type NodeValue added in v1.116.3 

type NodeValue func(node SelectedNode) float64

NodeValue returns a numerical value for each node.

func CreateNodeValue added in v1.116.3 

func CreateNodeValue(attr string) (NodeValue, error)

CreateNodeValue creates a NodeValue from a string definition.

type Nodes added in v1.122.1 

type Nodes struct {
	Name         string
	Groups       []*Nodes // if len(Groups) > 0, len(Nodes) == 0 and vice versa.
	Nodes        []*SelectedNode
	NodeSelector NodeSelector // nil if len(nodes) == 0
	Random       WeightedRandom
}

Nodes is a tree structure to store nodes and their attributes. Example: first level of the tree contains groups for tag:datacenter, second level contains groups for tag:servers, third level contains nodes. Selection on each level are predefined (example: select 3 datacenters, 2 servers from each datacenter, 1 node from each server). Selection is based on weight.

func (*Nodes) Add added in v1.122.1 

func (n *Nodes) Add(node *SelectedNode, attributes []NodeAttribute, weight float64)

Add adds a node to the tree. Based on the attributes (like datacenter,server,...) we build a tree. Note: weights are cumulative, on datacenter level, the weight is the sum of all the servers weights.

func (*Nodes) Select added in v1.122.1 

func (n *Nodes) Select(splits []int, m int, excluded []storj.NodeID) (selection []*SelectedNode)

Select selects nodes based on the selection pattern. parameter defines the desired number of groups from each level. Number of selected nodes will be the multiplication of all the nodes.

type NoopSuccessTracker added in v1.117.5 

type NoopSuccessTracker struct {
}

NoopSuccessTracker doesn't track uploads at all. Always returns with zero.

func (NoopSuccessTracker) Get added in v1.117.5 

func (n NoopSuccessTracker) Get(uplink storj.NodeID) func(node *SelectedNode) float64

Get implements UploadSuccessTracker.

type OrFilter added in v1.91.2 

type OrFilter []NodeFilter

OrFilter will include the node, if at lest one of the filters are matched.

func (OrFilter) Match added in v1.91.2 

func (n OrFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (OrFilter) String added in v1.91.2 

func (n OrFilter) String() string

type Placement added in v1.95.1 

type Placement struct {
	// the unique ID of the placement
	ID storj.PlacementConstraint
	// meaningful identifier/label for Humans. Will be used on UI.
	Name string
	// binding condition for filtering out nodes
	NodeFilter NodeFilter
	// binding condition for filtering out nodes, but only during uploads. Repair won't move nodes based on this.
	UploadFilter NodeFilter
	// Selector is the method how the nodes are selected from the full node space (eg. pick a subnet first, and pick a node from the subnet)
	Selector NodeSelectorInit
	// checked by repair job, applied to the full selection. Out of placement items will be replaced by new, selected by the Selector.
	Invariant Invariant
	// DownloadSelector is the method for how the nodes are selected for
	// downloading (e.g., at random from the uploaded set, or filtered down with
	// choice of 2).
	DownloadSelector DownloadSelector

	// CohortRequirements, if set, specify how the uplink will determine if
	// enough pieces with the right rules have been uploaded.
	CohortRequirements *CohortRequirements
	// CohortNames, if set, specifies how to calculate cohort names from a given
	// SelectedNode, for return in the AddressedOrderLimits.
	CohortNames map[string]CohortName

	// EC defines erasure coding parameter overrides.
	EC ECParameters `yaml:"ec"`
}

Placement defined all the custom behavior metadata of a specific placement group.

func (Placement) GetAnnotation deprecated added in v1.95.1 

func (p Placement) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

Deprecated: use Name instead.

func (Placement) Match added in v1.95.1 

func (p Placement) Match(node *SelectedNode) bool

Match implements NodeFilter.

func (Placement) MatchForUpload added in v1.125.2 

func (p Placement) MatchForUpload(node *SelectedNode) bool

MatchForUpload implements NodeFilter. It checks not just the global filter but also the filter for uploads.

type PlacementConfig added in v1.120.1 

type PlacementConfig struct {
	Placement ConfigurablePlacementRule `` /* 283-byte string literal not displayed */
}

PlacementConfig is placement configuration, separated from other generic configuration.

type PlacementConfigEnvironment added in v1.105.3 

type PlacementConfigEnvironment map[any]any

PlacementConfigEnvironment includes all generic functions and variables, which can be used in the configuration.

func NewPlacementConfigEnvironment added in v1.105.3 

func NewPlacementConfigEnvironment(successTracker UploadSuccessTracker, failureTracker UploadFailureTracker) PlacementConfigEnvironment

NewPlacementConfigEnvironment creates PlacementConfigEnvironment. It initializes the environment with successTracker, failureTracker, and any additional key-value pairs provided in additionalEnv.

func (PlacementConfigEnvironment) AddPrometheusTracker added in v1.129.3 

func (e PlacementConfigEnvironment) AddPrometheusTracker(tracker any) PlacementConfigEnvironment

AddPrometheusTracker adds a Prometheus tracker to the environment.

type PlacementDefinitions added in v1.95.1 

type PlacementDefinitions map[storj.PlacementConstraint]Placement

PlacementDefinitions can include the placement definitions for each known identifier.

func LoadConfig added in v1.96.2 

func LoadConfig(configFile string, environment PlacementConfigEnvironment) (PlacementDefinitions, error)

LoadConfig loads the placement yaml file and creates the Placement definitions.

func LoadConfigFromString added in v1.122.1 

func LoadConfigFromString(config string, environment PlacementConfigEnvironment) (PlacementDefinitions, error)

LoadConfigFromString loads the placement YAML from a string and creates the Placement definitions.

func NewPlacementDefinitions added in v1.95.1 

func NewPlacementDefinitions(placements ...Placement) PlacementDefinitions

NewPlacementDefinitions creates a PlacementDefinition with a default placement.

func TestPlacementDefinitions added in v1.96.2 

func TestPlacementDefinitions() PlacementDefinitions

TestPlacementDefinitions creates placements for testing. Only 0 placement is defined with subnetfiltering.

func TestPlacementDefinitionsWithFraction added in v1.96.2 

func TestPlacementDefinitionsWithFraction(newNodeFraction float64) PlacementDefinitions

TestPlacementDefinitionsWithFraction creates placements for testing. Similar to TestPlacementDefinitions, but also selects newNodes based on fraction.

func (PlacementDefinitions) AddLegacyStaticRules added in v1.95.1 

func (d PlacementDefinitions) AddLegacyStaticRules()

AddLegacyStaticRules initializes all the placement rules defined earlier in static golang code.

func (PlacementDefinitions) AddPlacement added in v1.96.2 

func (d PlacementDefinitions) AddPlacement(id storj.PlacementConstraint, placement Placement)

AddPlacement registers a new placement.

func (PlacementDefinitions) AddPlacementFromString deprecated added in v1.95.1 

func (d PlacementDefinitions) AddPlacementFromString(definitions string) error

AddPlacementFromString parses placement definition form string representations from id:definition;id:definition;...

Deprecated: we will switch to the YAML based configuration.

func (PlacementDefinitions) AddPlacementRule added in v1.95.1 

func (d PlacementDefinitions) AddPlacementRule(id storj.PlacementConstraint, filter NodeFilter, downloadSelector DownloadSelector)

AddPlacementRule registers a new placement.

func (PlacementDefinitions) CreateFilters added in v1.95.1 

func (d PlacementDefinitions) CreateFilters(constraint storj.PlacementConstraint) (filter NodeFilter, selector DownloadSelector)

CreateFilters implements PlacementCondition.

func (PlacementDefinitions) SupportedPlacements added in v1.95.1 

func (d PlacementDefinitions) SupportedPlacements() (res []storj.PlacementConstraint)

SupportedPlacements returns all the IDs, which have associated placement rules.

type PlacementRules added in v1.95.1 

type PlacementRules func(constraint storj.PlacementConstraint) (filter NodeFilter, selector DownloadSelector)

PlacementRules can crate filter based on the placement identifier.

type RandomOrder added in v1.86.1 

type RandomOrder struct {
	// contains filtered or unexported fields
}

RandomOrder as an iterator of a pseudo-random permutation set.

func NewRandomOrder added in v1.86.1 

func NewRandomOrder(n int) RandomOrder

NewRandomOrder creates new iterator, returns number between [0,n) in pseudo-random order.

func (*RandomOrder) At added in v1.86.1 

func (r *RandomOrder) At() uint64

At returns the current number in the permutations.

func (*RandomOrder) Finished added in v1.104.4 

func (r *RandomOrder) Finished() bool

Finished returns true, if there is no more permutation.

func (*RandomOrder) Next added in v1.86.1 

func (r *RandomOrder) Next() bool

Next generates the next number.

func (*RandomOrder) Reset added in v1.104.4 

func (r *RandomOrder) Reset()

Reset makes it possible to reuse the RandomOrder: the full pseudo-random permutations can be read again.

type ScoreNode added in v1.113.5 

type ScoreNode interface {
	Get(uplink storj.NodeID) func(node *SelectedNode) float64
}

ScoreNode can help to assign a score to a node. The higher score is better. float.Nan is valid, if no information is available.

func Desc added in v1.113.5 

func Desc(original ScoreNode) ScoreNode

Desc is a score node, which reverses the score of the original node.

func PieceCount added in v1.113.5 

func PieceCount(divider int64) ScoreNode

PieceCount scores the node based on the piece count.

type ScoreNodeFunc added in v1.113.5 

type ScoreNodeFunc func(uplink storj.NodeID, node *SelectedNode) float64

ScoreNodeFunc implements ScoreNode interface with a single func.

func (ScoreNodeFunc) Get added in v1.113.5 

func (s ScoreNodeFunc) Get(id storj.NodeID) func(node *SelectedNode) float64

Get implements ScoreNode.

type ScoreSelection added in v1.113.5 

type ScoreSelection func(uplink storj.NodeID, selected []*SelectedNode) float64

ScoreSelection can help to choose between two selections with assigning a score. The higher score is better.

func LastBut added in v1.113.5 

func LastBut(attr ScoreNode, skip int64) ScoreSelection

LastBut scores a selection based on the worst node (but skip the worst n nodes).

func MaxGroup added in v1.113.5 

func MaxGroup(attr NodeAttribute) ScoreSelection

MaxGroup returns with the size of the biggest group in the node selection.

func Median added in v1.113.5 

func Median(attr ScoreNode) ScoreSelection

Median scores the selection based on the median of the attribute.

type SelectedNode added in v1.84.1 

type SelectedNode struct {
	ID          storj.NodeID
	Address     *pb.NodeAddress
	Email       string
	Wallet      string
	LastNet     string
	LastIPPort  string
	CountryCode location.CountryCode
	Exiting     bool
	Suspended   bool
	Online      bool
	Vetted      bool
	Tags        NodeTags
	PieceCount  int64
	// free disk space in bytes
	FreeDisk int64
}

SelectedNode is used as a result for creating orders limits.

func (*SelectedNode) Clone added in v1.84.1 

func (node *SelectedNode) Clone() *SelectedNode

Clone returns a deep clone of the selected node.

type State 

type State map[storj.PlacementConstraint]NodeSelector

State includes a stateful selector (indexed nodes) for each placement.

func InitState added in v1.128.3 

func InitState(ctx context.Context, nodes []*SelectedNode, placements PlacementDefinitions) State

InitState initializes the State for each placement.

func (State) Select 

func (s State) Select(ctx context.Context, requester storj.NodeID, p storj.PlacementConstraint, count int, excluded []storj.NodeID, alreadySelected []*SelectedNode) (_ []*SelectedNode, err error)

Select picks the required nodes given a specific placement.

type StreamConstraint added in v1.126.4 

type StreamConstraint func([]*SelectedNode, *SelectedNode) bool

StreamConstraint is a function that determines if a node should be included in a stream. Returns true if the node should be included, false otherwise.

type TagFilter added in v1.84.1 

type TagFilter struct {
	// contains filtered or unexported fields
}

TagFilter matches nodes with specific tags.

func NewTagFilter added in v1.84.1 

func NewTagFilter(id storj.NodeID, name string, value []byte, match ValueMatch) TagFilter

NewTagFilter creates a new tag filter.

func (TagFilter) Match added in v1.88.2 

func (t TagFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (TagFilter) String added in v1.89.2 

func (t TagFilter) String() string

type UploadFailureTracker added in v1.117.5 

type UploadFailureTracker interface {
	Get(node *SelectedNode) float64
}

UploadFailureTracker keeps track of node failures.

type UploadSuccessTracker added in v1.105.3 

type UploadSuccessTracker interface {
	// Get gives a Score to the node based on the upload success rate. Can be math.NaN (no information). Higher is better.
	Get(uplink storj.NodeID) func(node *SelectedNode) float64
}

UploadSuccessTracker can give hints about the frequency of the long-tail cancellation per node.

type ValueMatch added in v1.89.2 

type ValueMatch func(a []byte, b []byte) bool

ValueMatch defines how to compare tag value with the defined one.

type WeightedItem added in v1.122.1 

type WeightedItem struct {
	Index int
	Score float64
}

WeightedItem is a helper struct for WeightedRandom. Includes the original index and randomized Score.

type WeightedRandom added in v1.122.1 

type WeightedRandom []float64

WeightedRandom provides random selection based on Efraimidis-Spirakis algorithm. see: http://utopia.duth.gr/~pefraimi/research/data/2007EncOfAlg.pdf

func (WeightedRandom) Random added in v1.122.1 

func (w WeightedRandom) Random(k int, exclusion []int) []int

Random returns k random indexes based on weights.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.