How to Retrieve a Non-Duplicate Advisory List?

[MaineK00n]

I used the following go program to page through the v2 advisories API:

• main.go

package main

import (
	"encoding/json"
	"fmt"
	"net/http"
	"time"
)

type advisories struct {
	Advisories  []advisory `json:"advisories"`
	LastUpdated time.Time  `json:"lastUpdated"`
	Page        int        `json:"page"`
	Size        int        `json:"size"`
	Total       int        `json:"total"`
}

type advisory struct {
	Name string `json:"name,omitempty"`
}

func main() {
	if err := run(); err != nil {
		panic(err)
	}
}

func run() error {
	u := "https://errata.build.resf.org/api/v2/advisories?page=%d&limit=25"

	for i := 0; ; i++ {
		as, err := func() ([]advisory, error) {
			resp, err := http.Get(fmt.Sprintf(u, i))
			if err != nil {
				return nil, err
			}
			defer resp.Body.Close()

			var advs advisories
			if err := json.NewDecoder(resp.Body).Decode(&advs); err != nil {
				return nil, err
			}

			return advs.Advisories, nil
		}()
		if err != nil {
			return err
		}
		if len(as) == 0 {
			break
		}

		for _, a := range as {
			fmt.Println(a.Name)
		}

		time.Sleep(1 * time.Second)
	}

	return nil
}

$ go run main.go | sort | uniq -c > advs.txt

$ cat advs.txt
...
      1 RLSA-2025:4051
      2 RLSA-2025:4063
      4 RLSA-2025:4170
      1 RLSA-2025:4244
      1 RLSA-2025:4263
      1 RLSA-2025:4341
      3 RLSA-2025:4362
      1 RLSA-2025:4443
      3 RLSA-2025:4458
      3 RLSA-2025:4459
      1 RLSA-2025:4460
      3 RLSA-2025:4461
      1 RLSA-2025:4487
...

$ cat advs.txt | wc -l
7400

At the point where I ran the command, the total number of advisories was 8014. (No additional advisories were added during this time.)
The actual number of unique advisories that I was able to retrieve by paging was 7400.
Looking at advs.txt, I noticed that many of the advisories appeared duplicated while paging.

How can I get all advisories without any duplicates?

[MaineK00n]

The v3 API does not appear to have this issue.

[atomicturtle]

Im working on improving all of this right now. Id love to discuss this with you on any other improvements we can make here. Im using vuls here too, its a great project.