Skip to content

Commit

Permalink
Merge pull request #4 from swayne275/readme-and-cleanup
Browse files Browse the repository at this point in the history
Cleanup retry/repeat, update readme
  • Loading branch information
swayne275 authored Nov 8, 2024
2 parents b941c3e + 903e1c4 commit 2f29e4b
Show file tree
Hide file tree
Showing 11 changed files with 624 additions and 477 deletions.
298 changes: 208 additions & 90 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,85 +5,37 @@
Builds off of the wonderful work of https://github.com/sethvargo/go-retry but adds additional functionality:

TODO:
- update godoc to my version
- update this documentation with changes
- add benchmarks of mine vs sethvargo (benchmark_test.go), and update the benchmarks section of the readme
- ensure godoc up to date

## Added features
## Synopsis

### infinite retries until error
Repeat will continually do whatever is in the RepeatFunc until it returns an error.

It does not observe RetryableError, as there is no need for this functionality.

### backoff reset
You might want to reset a backoff of non constant duration (eg if an activity happens that
says you should poll faster).

Retry is a Go library for facilitating retry logic and backoff. It's highly
extensible with full control over how and when retries occur. You can also write
your own custom backoff functions by implementing the Backoff interface.
Retry is a Go library for facilitating retry logic and backoff strategies. It builds off the work of [go-retry](https://github.com/sethvargo/go-retry) and adds additional functionality, such as infinite retries until an error occurs and the ability to reset backoff durations. The library is highly extensible, allowing you to implement custom backoff strategies and integrate them seamlessly into your applications.

## Features

- **Extensible** - Inspired by Go's built-in HTTP package, this Go backoff and
retry library is extensible via middleware. You can write custom backoff
functions or use a provided filter.

- **Independent** - No external dependencies besides the Go standard library,
meaning it won't bloat your project.

- **Concurrent** - Unless otherwise specified, everything is safe for concurrent
use.

- **Context-aware** - Use native Go contexts to control cancellation.

## Usage

Here is an example use for connecting to a database using Go's `database/sql`
package:
- **Infinite Retries Until Error** - Repeat will continually do whatever is in the RepeatFunc until it returns an error. It does not observe RetryableError, as there is no need for this functionality.

```golang
package main
- **Backoff Reset** - You might want to reset a backoff of non-constant duration (e.g., if something happens that says you should poll faster, like a worker having a job available).

import (
"context"
"database/sql"
"log"
"time"
- **Extensible** - Inspired by Go's built-in HTTP package, this Go backoff and retry library is extensible via middleware. You can write custom backoff functions or use a provided filter.

"github.com/swayne275/go-retry"
)
- **Independent** - No external dependencies besides the Go standard library, meaning it won't bloat your project.

func main() {
db, err := sql.Open("mysql", "...")
if err != nil {
log.Fatal(err)
}

ctx := context.Background()
if err := retry.Fibonacci(ctx, 1*time.Second, func(ctx context.Context) error {
if err := db.PingContext(ctx); err != nil {
// This marks the error as retryable
return retry.RetryableError(err)
}
return nil
}); err != nil {
log.Fatal(err)
}
}
```
- **Concurrent** - Unless otherwise specified, everything is safe for concurrent use.

## Backoffs
### Backoff Strategies

In addition to your own custom algorithms, there are built-in algorithms for
backoff in the library.

### Constant
#### Constant Backoff
Retries at a constant interval.

A very rudimentary backoff, just returns a constant value. Here is an example:
Example:

```text
1s -> 1s -> 1s -> 1s -> 1s -> 1s
1s -> 1s -> 1s -> 1s -> 1s -> 1s -> 1s
```

Usage:
Expand All @@ -92,10 +44,12 @@ Usage:
NewConstant(1 * time.Second)
```

### Exponential
#### Exponential Backoff
Retries with exponentially increasing intervals.

Arguably the most common backoff, the next value is double the previous value.
Here is an example:

Example:

```text
1s -> 2s -> 4s -> 8s -> 16s -> 32s -> 64s
Expand All @@ -107,15 +61,16 @@ Usage:
NewExponential(1 * time.Second)
```

### Fibonacci
#### Fibonacci Backoff
Retries with intervals following the Fibonacci sequence.

The next value is the sum of the current value and the previous value. This means retires happen quickly at first, but then gradually take slower, ideal for
network-type issues.

The Fibonacci backoff uses the Fibonacci sequence to calculate the backoff. The
next value is the sum of the current value and the previous value. This means
retires happen quickly at first, but then gradually take slower, ideal for
network-type issues. Here is an example:
Example:

```text
1s -> 1s -> 2s -> 3s -> 5s -> 8s -> 13s
1s -> 2s -> 3s -> 5s -> 8s -> 13s
```

Usage:
Expand All @@ -124,61 +79,224 @@ Usage:
NewFibonacci(1 * time.Second)
```

## Modifiers (Middleware)
### Modifiers (Middleware)

The built-in backoff algorithms never terminate and have no caps or limits - you
control their behavior with middleware. There's built-in middleware, but you can
also write custom middleware.

### Jitter
#### Jitter
Adds randomness to the backoff intervals to prevent thundering herd problems.

To reduce the changes of a thundering herd, add random jitter to the returned
To reduce the chances of a thundering herd, add random jitter to the returned
value.

```golang
b, err := NewFibonacci(1 * time.Second)
backoff, err := NewFibonacci(1 * time.Second)

// Return the next value, +/- 500ms
b, err = WithJitter(500*time.Millisecond, b)
backoffWithJitter, err := WithJitter(500*time.Millisecond, backoff)

// Return the next value, +/- 5% of the result
b, err = WithJitterPercent(5, b)
backoffWithJitterPercent, err := WithJitterPercent(5, backoff)
```

#### Capped Duration
Limits the maximum duration between retries.

To ensure an individual calculated duration never exceeds a value, use a cap:

```golang
backoff, err := NewFibonacci(1 * time.Second)

// Ensure the maximum value is 2s. In this example, the sleep values would be
// 1s, 1s, 2s, 2s, 2s, 2s...
backoffWithCap := WithCappedDuration(2 * time.Second, backoff)
```

#### Max Duration
Limits the maximum total time a backoff should execute.

For a best-effort limit on the total execution time, specify a max duration:

```golang
backoff, err := NewFibonacci(1 * time.Second)

// Ensure the maximum total retry time is 5s.
backoffWithMaxDuration = WithMaxDuration(5 * time.Second, backoff)
```

### MaxRetries
#### Max Retries
Limits the number of retry attempts.

To terminate a retry, specify the maximum number of _retries_. Note this
is _retries_, not _attempts_. Attempts is retries + 1.

```golang
b, err := NewFibonacci(1 * time.Second)
backoff, err := NewFibonacci(1 * time.Second)

// Stop after 4 retries, when the 5th attempt has failed. In this example, the worst case elapsed
// time would be 1s + 1s + 2s + 3s = 7s.
b = WithMaxRetries(4, b)
backoffWithMaxRetries = WithMaxRetries(4, backoff)
```

### CappedDuration
#### Context-Aware Backoff
Stops the backoff if the provided context is Done.

To ensure an individual calculated duration never exceeds a value, use a cap:
```golang
backoff, err := NewFibonacci(1 * time.Second)
ctx, cancel := context.WithTimeout(context.Background, 1 * time.Millisecond)

// backoff will return stop == true when context is cancelled
backoffWithContext := WithContext(ctx, backoff)
```

## Installation

To install the library, use the following command:

```sh
go get github.com/swayne275/go-retry
```

## Usage

### Basic Retry

This will retry the provided function until it either succeeds or returns a non-retryable error.

```golang
b, err := NewFibonacci(1 * time.Second)
package main

// Ensure the maximum value is 2s. In this example, the sleep values would be
// 1s, 1s, 2s, 2s, 2s, 2s...
b = WithCappedDuration(2 * time.Second, b)
import (
"context"
"fmt"
"time"

"github.com/swayne275/go-retry"
"github.com/swayne275/go-retry/backoff"
)

func main() {
ctx := context.Background()
backoff := backoff.NewConstant(1 * time.Second)

err := retry.Do(ctx, backoff, func(ctx context.Context) error {
// Your retryable function logic here
return nil
})

if err != nil {
fmt.Printf("Operation failed: %v\n", err)
}
}
```

### WithMaxDuration
### Infinite Repeat Until Non Retryable Error

For a best-effort limit on the total execution time, specify a max duration:
This will repeat the function until it returns a non-retryable error.

```golang
b, err := NewFibonacci(1 * time.Second)
package main

// Ensure the maximum total retry time is 5s.
b = WithMaxDuration(5 * time.Second, b)
import (
"context"
"fmt"

"github.com/swayne275/go-retry/repeat"
)

func main() {
ctx := context.Background()
backoff := backoff.NewExponential(1 * time.Second)

err := repeat.Do(ctx, backoff, func(ctx context.Context) bool {
// Your function logic here - return false to stop repeating
return true
})

if err != nil {
// you can check why the repeat stopped with errors.Is() and the defined
// types in the repeat package
fmt.Printf("Operation failed: %v\n", err)
}
}
```

### Backoff Reset

```golang
package main

import (
"context"
"fmt"
"time"

"github.com/swayne275/go-retry"
"github.com/swayne275/go-retry/backoff"
)

func main() {
ctx := context.Background()
backoff := backoff.NewExponential(1 * time.Second)
resetFunc := func() Backoff {
// define how the backoff shoudl be reset, likely something like:
newB, err := NewExponential(base)
if err != nil {
t.Fatalf("failed to reset exponential backoff: %v", err)
}

return newB
}
backoffWithReset = backoff.WithReset(resetFunc, backoff)

err := retry.Do(ctx, backoffWithReset, func(ctx context.Context) error {
// Your retryable function logic here

// something happens that makes you want to reset back to a shorter backoff
b.Reset()

return nil
})

if err != nil {
fmt.Printf("Operation failed: %v\n", err)
}
}
```

### Real World Example: Connecting to a SQL Database

```golang
package main

import (
"context"
"database/sql"
"log"
"time"

"github.com/swayne275/go-retry"
)

func main() {
db, err := sql.Open("mysql", "...")
if err != nil {
log.Fatal(err)
}

ctx := context.Background()
if err := retry.FibonacciRetry(ctx, 1*time.Second, func(ctx context.Context) error {
if err := db.PingContext(ctx); err != nil {
// This marks the error as retryable
return retry.RetryableError(err)
}
return nil
}); err != nil {
log.Fatal(err)
}
}
```

## Benchmarks
Expand Down
2 changes: 0 additions & 2 deletions backoff/backoff.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ import (
"github.com/swayne275/go-retry/internal/random"
)

// TODO clean up interface, struct, etc

// Backoff is an interface that backs off.
type Backoff interface {
// Next returns the time duration to wait and whether to stop.
Expand Down
Loading

0 comments on commit 2f29e4b

Please sign in to comment.