Control the retry policy

Using Golem's retry mechanism

Golem applies a retry mechanism to all agents. In case of a failure, Golem will automatically recover the agent to the point before the failure and retry the operation. An exponential backoff and an upper limit on the number of retries are applied.

If the maximum number of retries is reached, the agent will be marked as failed and no further invocations will be possible on it.

This mechanism is automatic and applied to all kinds of failures. To rely on it, throw an unhandled exception.

⚠️

Golem will retry from the point the unhandled exception was thrown. It is possible that a previous operation's unwanted result (if it did not end in an exception) has been already persisted, in which case it won't be retried.

Customizing the retry policy

The retry policy which controls the maximum number of retries and the exponential backoff is a global configuration of the Golem servers, but it can be customized for each agent.

The @golemcloud/golem-ts-sdk package exports the withRetryPolicy function to temporarily install a named retry policy for a block of code:

import {withRetryPolicy, type NamedRetryPolicy} from "@golemcloud/golem-ts-sdk"
 
// Durations are expected as nanoseconds
const policy: NamedRetryPolicy = {
  name: "temporary-policy",
  priority: 1000,
  predicate: {
    nodes: [{tag: "pred-true"}],
  },
  policy: {
    nodes: [
      {
        tag: "count-box",
        val: {
          maxRetries: 3,
          inner: 1,
        },
      },
      {
        tag: "clamp-delay",
        val: {
          minDelay: 100_000_000n, // 100 milliseconds
          maxDelay: 2_000_000_000n, // 2 seconds
          inner: 2,
        },
      },
      {
        tag: "exponential",
        val: {
          baseDelay: 100_000_000n,
          factor: 1.5,
        },
      },
    ],
  },
}
 
const result: string = withRetryPolicy(policy, () => {
  // this closure runs with the custom retry policy
  return "hello"
})

The policy passed to withRetryPolicy is a NamedRetryPolicy, which contains a name, priority, predicate, and policy tree:

export type RetryPredicate = {
  nodes: PredicateNode[];
};
 
export type RetryPolicy = {
  nodes: PolicyNode[];
};
 
export type NamedRetryPolicy = {
  name: string;
  priority: number;
  predicate: RetryPredicate;
  policy: RetryPolicy;
};

MoonBit uses a simpler RetryPolicy struct with max_attempts, min_delay, max_delay, multiplier, and max_jitter_factor fields, rather than the tree-based NamedRetryPolicy used by TypeScript, Rust, and Scala. The named retry policy system described below applies to TypeScript, Rust, and Scala only.

Named retry policies in golem.yaml

Retry policies can also be defined declaratively in the application manifest (golem.yaml). These named retry policies are created in the environment during deployment, making them available to all agents without requiring code changes.

retryPolicyDefaults:
  local:
    - name: default-retry
      priority: 10
      predicate: "true"
      policy:
        countBox:
          maxRetries: 3
          inner:
            exponential:
              baseDelay: { secs: 1, nanos: 0 }
              factor: 2.0

Policy types reference

The following policy types can be used to construct retry policy trees:

Type	Fields	Description
`"immediate"`	—	Retry immediately
`"never"`	—	Never retry
`periodic`	`{ secs, nanos }`	Fixed delay between retries
`exponential`	`{ baseDelay, factor }`	Exponentially increasing delay
`fibonacci`	`{ first, second }`	Fibonacci-sequence delays
`countBox`	`{ maxRetries, inner }`	Limit total retry count
`timeBox`	`{ limit, inner }`	Limit total retry time
`clamp`	`{ minDelay, maxDelay, inner }`	Clamp delay range
`addDelay`	`{ delay, inner }`	Add fixed delay to inner policy
`jitter`	`{ factor, inner }`	Add random jitter
`filteredOn`	`{ predicate, inner }`	Apply only when predicate matches
`andThen`	`[policy1, policy2]`	Sequential composition
`union`	`[policy1, policy2]`	Union of two policies
`intersect`	`[policy1, policy2]`	Intersection of two policies

Predicates reference

Predicates control when a retry policy is applied. The following predicate types are available:

Type	Fields	Description
`"true"`	—	Always matches
`"false"`	—	Never matches
`propEq`	`{ property, value }`	Property equals value
`propNeq`	`{ property, value }`	Property not equal
`propGt` / `propGte`	`{ property, value }`	Greater than / greater or equal
`propLt` / `propLte`	`{ property, value }`	Less than / less or equal
`propExists`	string	Property exists
`propIn`	`{ property, values }`	Property in set
`propMatches`	`{ property, pattern }`	Regex match
`propStartsWith`	`{ property, prefix }`	Starts with prefix
`propContains`	`{ property, substring }`	Contains substring
`and`	`[pred1, pred2]`	Logical AND
`or`	`[pred1, pred2]`	Logical OR
`not`	predicate	Logical NOT

Predicate values are typed: { text: "..." }, { integer: 42 }, or { boolean: true }.

Snapshotting Transactions