gotosocial/vendor/codeberg.org/gruf/go-structr
kim c9c0773f2c
[performance] update remaining worker pools to use queues (#2865)
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
..
cache.go [performance] update remaining worker pools to use queues (#2865) 2024-04-26 13:50:46 +01:00
index.go [chore] update go-structr => v0.6.2 (fixes nested field ptr following) (#2822) 2024-04-11 11:46:08 +02:00
item.go [chore] bump go structr cache version -> v0.6.0 (#2773) 2024-04-02 12:03:40 +02:00
key.go [chore] bump go structr cache version -> v0.6.0 (#2773) 2024-04-02 12:03:40 +02:00
LICENSE [performance] overhaul struct (+ result) caching library for simplicity, performance and multiple-result lookups (#2535) 2024-01-19 12:57:29 +00:00
list.go [chore] bump go structr cache version -> v0.6.0 (#2773) 2024-04-02 12:03:40 +02:00
queue.go [chore] update go-structr => v0.6.2 (fixes nested field ptr following) (#2822) 2024-04-11 11:46:08 +02:00
queue_ctx.go [performance] update remaining worker pools to use queues (#2865) 2024-04-26 13:50:46 +01:00
README.md [chore] bump go structr cache version -> v0.6.0 (#2773) 2024-04-02 12:03:40 +02:00
runtime.go [chore] update go-structr => v0.6.2 (fixes nested field ptr following) (#2822) 2024-04-11 11:46:08 +02:00
test.sh update go-structr v0.2.0 => v0.3.0 to fix possible hash collision issues (#2586) 2024-01-29 15:13:53 +00:00
util.go [performance] update remaining worker pools to use queues (#2865) 2024-04-26 13:50:46 +01:00

go-structr

A library with a series of performant data types with automated struct value indexing. Indexing is supported via arbitrary combinations of fields, and in the case of the cache type, negative results (errors!) are also supported.

Under the hood, go-structr maintains a hashmap per index, where each hashmap is a hashmap keyed by serialized input key type. This is handled by the incredibly performant serialization library go-mangler, which at this point in time supports just about any arbitrary type, so feel free to index by anything!

Cache example

type Cached struct {
    Username    string
    Domain      string
    URL         string
    CountryCode int
}

var c structr.Cache[*Cached]

c.Init(structr.CacheConfig[*Cached]{

    // Fields this cached struct type
    // will be indexed and stored under.
    Indices: []structr.IndexConfig{
        {Fields: "Username,Domain", AllowZero: true},
        {Fields: "URL"},
        {Fields: "CountryCode", Multiple: true},
    },

    // Maximum LRU cache size before
    // new entries cause evictions.
    MaxSize: 1000,

    // User provided value copy function to
    // reduce need for reflection + ensure
    // concurrency safety for returned values.
    Copy: func(c *Cached) *Cached {
        c2 := new(Cached)
        *c2 = *c
        return c2
    },

    // User defined invalidation hook.
    Invalidate: func(c *Cached) {
        log.Println("invalidated:", c)
    },
})

// Access and store indexes ahead-of-time for perf.
usernameDomainIndex := c.Index("Username,Domain")
urlIndex := c.Index("URL")
countryCodeIndex := c.Index("CountryCode")

var url string

// Generate URL index key.
urlKey := urlIndex.Key(url)

// Load value from cache, with callback function to hydrate
// cache if value cannot be found under index name with key.
// Negative (error) results are also cached, with user definable
// errors to ignore from caching (e.g. context deadline errs).
value, err := c.LoadOne(urlIndex, func() (*Cached, error) {
    return dbType.SelectByURL(url)
}, urlKey)
if err != nil {
    return nil, err
}

// Store value in cache, only if provided callback
// function returns without error. Passes value through
// invalidation hook regardless of error return value.
//
// On success value will be automatically added to and
// accessible under all initially configured indices.
if err := c.Store(value, func() error {
    return dbType.Insert(value)
}); err != nil {
    return nil, err
}

// Generate country code index key.
countryCodeKey := countryCodeIndex.Key(42)

// Invalidate all cached results stored under
// provided index name with give field value(s).
c.Invalidate(countryCodeIndex, countryCodeKey)

Queue example


type Queued struct{
    Username    string
    Domain      string
    URL         string
    CountryCode int
}

var q structr.Queue[*Queued]

q.Init(structr.QueueConfig[*Cached]{

    // Fields this queued struct type
    // will be indexed and stored under.
    Indices: []structr.IndexConfig{
        {Fields: "Username,Domain", AllowZero: true},
        {Fields: "URL"},
        {Fields: "CountryCode", Multiple: true},
    },

    // User defined pop hook.
    Pop: func(c *Cached) {
        log.Println("popped:", c)
    },
})

// Access and store indexes ahead-of-time for perf.
usernameDomainIndex := q.Index("Username,Domain")
urlIndex := q.Index("URL")
countryCodeIndex := q.Index("CountryCode")

// ...
q.PushBack(Queued{
    Username:   "billybob",
    Domain:     "google.com",
    URL:        "https://some.website.here",
    CountryCode: 42,
})

// ...
queued, ok := q.PopFront()

// Generate country code index key.
countryCodeKey := countryCodeIndex.Key(42)

// ...
queuedByCountry := q.Pop(countryCodeIndex, countryCodeKey)

Notes

This is a core underpinning of GoToSocial's performance.