* implement status pin client api + web handler
* make test names + comments more descriptive
* don't use separate table for status pins
* remove unused add + remove checking
* tidy up + add some more tests
This changes parseDescription to properly encode things to be safe for
usage without removing things like backslashes that may be relevant.
* text.SanitizePlaintext already calls html.UnescapeString so we don't
have to do that
* Replace \n with space early
* Remove duplicate white-space by splitting on fields and joining
* HTML-escape the string we have
* For extra certainty, encode the backslash as \
Fixes#1549
* Tidy up emoji parsing on profile submission
Don't bother reparsing for emoji unless one of the fields that can have
emoji in it has changed.
Deduplicate emoji between the display name and profile note - I'm not
sure whether this was hurting anything, but better safe.
* Deduplicate emoji when parsing remote accounts
Some servers - Misskey at least - don't deduplicate emoji, so it's
possible to get an account which has the same emoji used in both the
display name and note and therefore includes that emoji twice in its
metadata. When we start trying to put those into our database, we run
into a uniqueness constraint and fall over.
This change just deduplicates at the point of construction of an
account.
This moves checking if the request is valid as early as possible in the
chain. This should ensure that for an invalid request we never bother
acquiring the wait queue and taking up a spot in it.
* improve error readability, mark "bad hosts" as fastFail
Signed-off-by: kim <grufwub@gmail.com>
* pull in latest go-byteutil version with byteutil.Reader{}
Signed-off-by: kim <grufwub@gmail.com>
* use rewindable body reader for post requests
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
This updates the middleware log.WithField calls that create new loggers
to include the context the first time around. Without it the requestID
does not get logged.
Fixup from #1476
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.
Co-authored-by: kim <grufwub@gmail.com>
* [bugfix] set cache-control max-age dynamically for s3
* woops
* double whoops
* time until, thank you linter, bless you, you're the best, no matter what kim says
* aa
* start fiddling with media + account queries a little
* initialize state when pruning
* allow for unsetting remote media
make sure to wait til media loaded
fix silly tiny bug
* move comment a bit for readability
* slight reformat of fetchRemoteAccount{Avatar,Header}
* fix issue after rebase
* slightly neaten up logic of avatar/header media handling
* remove log prefix (callername log field handles this)
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling
* improved code comment
* fix worker tryUntil function, update go-runners/go-sched
* make preprocess functions package public, use these where possible to stop doubled up processing
* remove separate emoji worker pool
* limit calls to time.Now() during media preprocessing
* use Processor{} to manage singular runtime of processing media
* ensure workers get started when media manager is used
* improved error setting in processing media, fix media test
* port changes from processingmedia to processing emoji
* finish code commenting
* finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced
* linterrrrrrrrrrrrrrrr
---------
Signed-off-by: kim <grufwub@gmail.com>
* remove throttling timers, support setting retry-after, use retry-after in transport
* remove unused variables
* add throttling-retry-after to cmd flags
* update envparsing to include new throttling-retry-after
* update example config to include retry-after documentation
* also support retry-after formatted as date-time, ensure max backoff time
---------
Signed-off-by: kim <grufwub@gmail.com>
* small formatting changes, rewrite fetchRemoteMedia to use separate funcs + use mutex lock correctly
* move url parsing before acquiring mutex locks
* use wrapped mutexes to allow safe unlocking. (previously i did a fucky and passed mutex by value...)
* remove unused code
* use consistent map keying for dereferencing headers/avatars
---------
Signed-off-by: kim <grufwub@gmail.com>
* serve publickey separately from AP, don't throttle it
* update nginx cache documentation, cache main-key too
* throttle public key, but separately from other endpoints
* remove local copying of file for satisfying range headers
Signed-off-by: kim <grufwub@gmail.com>
* remove unused import
Signed-off-by: kim <grufwub@gmail.com>
* fix range header reader limit calculation
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* reformat GetAccount() functionality, and add UpdateAccount() function.
* use fetched_at instead of last_webfingered_at
* catch local "not found" errors. small formatting / error string changes
* remove now unused error type
* return nil when wrapping nil error
* update expected error messages
* return correct url for foss satan webfinger
* add AP model for Some_User
* normalize local domain
* return notretrievable where appropriate
* expose NewErrNotRetrievable
* ensure webfinger for new accounts searched by uri
* update local account short circuit
* allow enrich to fail for already-known accounts
* remove unused LastWebfingeredAt
* expose test maps on mock http client
* update Update test
* reformat GetAccount() functionality, and add UpdateAccount() function.
* use fetched_at instead of last_webfingered_at
* catch local "not found" errors. small formatting / error string changes
* remove nil error checks (we shouldn't be passing nil errors to newError() initializers)
* remove mutex unlock on transport init fail (it hasn't yet been locked!)
* woops add back the error wrapping to use ErrNotRetrievable
* caches were never being started... 🙈
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* Implement goldmark debug print for hashtags and mentions
* Minify HTML in FromPlain
* Convert plaintext status parser to goldmark
* Move mention/tag/emoji finding logic into formatter
* Combine mention and hashtag boundary characters
* Normalize unicode when rendering hashtags
* [chore] set max open / idle conns + conn max lifetime for both postgres and sqlite
* reduce cache size default to 8MiB, reduce connections to 2 * cpu
* introduce max open conns multiplier, tune sqlite and pg separately
* go fmt
* start adding report client api
* route + test reports get
* start report create endpoint
* you can create reports now babyy
* stub account report processor
* add single reportGet endpoint
* fix test
* add more filtering params to /api/v1/reports GET
* update swagger
* use marshalIndent in tests
* add + test missing Link info
* don't serve unused fields for video attachments
* parse video bitrate + duration more accurately
* use ServeContent where appropriate to respect Range
* abstract temp file seeker into its own function
* media processor consolidation and reformatting, reduce amount of required syscalls
Signed-off-by: kim <grufwub@gmail.com>
* update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling
Signed-off-by: kim <grufwub@gmail.com>
* fix duration not being set, fix mp4 test expecting error
Signed-off-by: kim <grufwub@gmail.com>
* fix test expecting media files with different extension
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code
Signed-off-by: kim <grufwub@gmail.com>
* fix expected storage paths in tests, update expected test thumbnails
Signed-off-by: kim <grufwub@gmail.com>
* remove dead code
Signed-off-by: kim <grufwub@gmail.com>
* fix cached presigned s3 url fetching
Signed-off-by: kim <grufwub@gmail.com>
* fix tests
Signed-off-by: kim <grufwub@gmail.com>
* fix test models
Signed-off-by: kim <grufwub@gmail.com>
* update media processing to use sync.Once{} for concurrency protection
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* fix passing in KVStore GetStream() as stream to PutStream()
Signed-off-by: kim <grufwub@gmail.com>
* fix unlocks of storage keys
Signed-off-by: kim <grufwub@gmail.com>
* whoops, return the error...
Signed-off-by: kim <grufwub@gmail.com>
* pour one out for tobi's code <3
Signed-off-by: kim <grufwub@gmail.com>
* add back the byte slurping code
Signed-off-by: kim <grufwub@gmail.com>
* check for both ErrUnexpectedEOF and EOF
Signed-off-by: kim <grufwub@gmail.com>
* add back links to file format header information
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* Add local user and post count to nodeinfo responses
This fixes#1307 (at least partially). The nodeinfo endpoint should now
return the total users on an instance, along with their post count.
* Update NodeInfoUsers docstring and swagger yaml file
* launch websocket streaming in goroutine to allow upgrade handler to return
* don't send any message on ping, improved close check on failed read
* use context to signal wsconn close, ensure canceled in read goroutine
Signed-off-by: kim <grufwub@gmail.com>
* [feature] Add throttling middleware to AP endpoints
* refactor a lil bit
* use config setting, start updating docs
* doc updates
* use relative links in faq doc
* small docs fixes
* return code 503 instead of 429 when throttled
* throttle other endpoints too
* simplify token channel prefills
* interim commit: start refactoring middlewares into package under router
* another interim commit, this is becoming a big job
* another fucking massive interim commit
* refactor bookmarks to new style
* ambassador, wiz zeze commits you are spoiling uz
* she compiles, we're getting there
* we're just normal men; we're just innocent men
* apiutil
* whoopsie
* i'm glad noone reads commit msgs haha :blob_sweat:
* use that weirdo go-bytesize library for maxMultipartMemory
* fix media module paths
* start messing about with different mp4 metadata extraction
* heyyooo it works
* add test cow
* move useful multierror to gtserror package
* error out if video doesn't seem to be a real mp4
* test parsing mkv in disguise as mp4
* tidy up error handling
* remove extraneous line
* update framerate formatting
* use float32 for aspect
* fixy mctesterson
* close pipereader on failed data function
* gently slurp the bytes
* readability updates
* go fmt
* tidy up file server tests + add more cases
* start moving io wrappers to separate iotools package. Remove use of buffering while piping recache stream
Signed-off-by: kim <grufwub@gmail.com>
* add license text
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* for domain block lookups, lookup along subdomain parts
Signed-off-by: kim <grufwub@gmail.com>
* only lookup up to a max of 5 domain parts to prevent DOS, limit inserted domains to max of 5 subdomains
Signed-off-by: kim <grufwub@gmail.com>
* add test for domain block wildcarding
Signed-off-by: kim <grufwub@gmail.com>
* check cached status first, increase cached domain time
Signed-off-by: kim <grufwub@gmail.com>
* fix domain wildcard part building logic
Signed-off-by: kim <grufwub@gmail.com>
* create separate domain.BlockCache{} type to hold all domain blocks in memory
Signed-off-by: kim <grufwub@gmail.com>
* remove unused variable
Signed-off-by: kim <grufwub@gmail.com>
* add docs and test to domain block cache, check for domain == host in domain block getter funcs
Signed-off-by: kim <grufwub@gmail.com>
* add license text
Signed-off-by: kim <grufwub@gmail.com>
* check order in which we check primary cache
Signed-off-by: kim <grufwub@gmail.com>
* add better documentation of how domain block checking is performed
Signed-off-by: kim <grufwub@gmail.com>
* change
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* simplify pruneRemote, remove unncecessary media trace logging, update RemoteOlderThan() to include headers/avis
Signed-off-by: kim <grufwub@gmail.com>
* cleanup pruneallmeta, add remote header to pruneremote tests
Signed-off-by: kim <grufwub@gmail.com>
* fix olderthan duration additions
Signed-off-by: kim <grufwub@gmail.com>
* fix broken test now that test model header changed
Signed-off-by: kim <grufwub@gmail.com>
* instead use new remote test account for new header model
Signed-off-by: kim <grufwub@gmail.com>
* use newer generated ULID for remote_account_3 to ensure it is sorted last
Signed-off-by: kim <grufwub@gmail.com>
* reorganize serialized keys to match expected test account model order
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* update config generator to support nested structs, add cache configuration options
* update envparsing test
* add cache configuration to config parse tests
* set cache configuration in testrig
* move caches to sub-cache "gts" namespace, update envparsing, add cache config docs to example config
Signed-off-by: kim <grufwub@gmail.com>
* [chore] Move ShortcodeDomain to its own little util func
* [feature] Add RefetchEmojis function to media manager
* [feature] Expose admin media refresh via admin API
* update following review feedback
- change/fix log levels
- make sure not to try to refetch local emojis
- small style refactoring + comments
* log on emoji refetch start
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* Implement Bookmarks
* Update based on review comments
* Update swagger doc
* Fix argument passing to status.Bookmark
* Update changed test
* Updates based on latest PR review
* move caches to a separate State{} structure
Signed-off-by: kim <grufwub@gmail.com>
* fix call to log.Panic not using formatted call
Signed-off-by: kim <grufwub@gmail.com>
* move caches to use interfaces, to make switchouts easier in future
Signed-off-by: kim <grufwub@gmail.com>
* fix rebase issue
Signed-off-by: kim <grufwub@gmail.com>
* improve code comment
Signed-off-by: kim <grufwub@gmail.com>
* fix further issues after rebase
Signed-off-by: kim <grufwub@gmail.com>
* heh
Signed-off-by: kim <grufwub@gmail.com>
* add missing license text
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* [feature] overhaul the oidc system
this allows for more flexible username handling and prevents account
takeover using old email addresses
* [feature] add migration path for old OIDC users
* [feature] nicer error reporting for users
* [docs] document the new OIDC flow
* [fix] return early on oidc error
* [docs]: add comments on the finalization logic
Implements #864 and should speed up s3 based installations by a lot.
With more static urls, we can then also implement #1026 for even
better performance when used in conjunction with CDNs
In the previous changes that expanded the IPv4 and IPv6 deny lists based
on the IANA registries we inadvertently added a number of duplicates.
This is unnecessary as they're already caught by larger prefixes and
means there's less entries to scan.
This change removes all prefixes that are subnets of other prefixes.
* ap: add support for PKCS1 "RSA PUBLIC KEY" pem block type
Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
* ap: report no PEM data or unknown pem block type
Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
* only return error for emoji fetch if NOT errnoentries
Signed-off-by: kim <grufwub@gmail.com>
* reformat gts->api model slice conversion to standard error behaviours and reduce code reuse
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* start refactoring some of the search + deref logic
* add tests for search api
* rename GetRemoteAccount + GetRemoteStatus
* make search function a bit simpler + clearer
* fix little fucky wucky uwu owo i'm just a little guy
* update faulty switch statements
* update test to use storage struct
* redo switches for clarity
* reduce repeated logic in search tests
* fastfail getstatus by uri
* debug log + trace log better
* add implementation note
* return early if no result for namestring search
* return + check on dereferencing error types
* errors hah what errors
* remove unneeded error type alias, add custom error text during stringification itself
* fix a woops recursion 🙈
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
In d6f4d196c9 we swapped to use the
AccountDomain but that actually goes against the intent of the change.
This reverts that change and uses the host domain again.
Currently requests set their own User-Agent. This moves it down to set
it in the transport's do() method, to guarantee it's always set on all
requests.
* [bugfix] Use AccountDomain for user agent
By using the account domain we can pinpoint the source of the request
more accurately when looking at the User-Agent header.
* [chore] Align user-agent header with spec
Based on RFC 7231, our User-Agent header doesn't quite match. It seems
to always want Name [/ Version] pairs, with comments in parenthesis and
multiple comments separated by a semicolon.
Align our UA with that, using application name first by itself in case
someone has customised it with the source instance in a comment. Follow
that up with gotosocial/<version> and a comment pointing at the source
code.
This also drops the mention of gofed/activity since a fork is in use.
* [bugfix] Ensure requests happen over TCP
It's possible for the network to be udp4 or udp6. This is rather
unlikely to occur, but since we're given the network anyway as part of
the Sanitize function getting called we might as well check for it.
* [chore] Align reserved v6 blocks to IANA registry
* [chore] Add test for ValidateIP
The net and netip packages diverge in that net.ParseIP will consider an
IPv4-mapped address to be an IPv4 address and as such it would get
caught by the IPv4Reserved list. However, netip considers it an IPv6
address, so we need to ensure the mapped range is in IPv6Reserved.
* [chore] Align reserved v4 blocks to IANA registry
This includes a number of tests for /32's explicitly called out in the
registry to ensure we always consider those invalid.
* [bugfix]: Fix IPv6 validation
The current code considers ff00::/8 valid, but contrary to the comment
that's not the global unicast range. ff-prefixes in IPv6 denote
multicast.
This adapts the code to take the same approach as IPv4, explicitly
blacklisting reserved internal/private ranges.
* [chore] Add missing 4 in IPv4Reserved doc comment
* add FilePath regex
* add `admin media prune orphaned` command
* add prune orphaned function to media manager
* don't mark flag as required
* document admin media prune orphaned cmd
* oh envparsing.sh you coy minx
* start adding admin emoji PATCH stuff
* updating works OK, now how about copying
* allow emojis to be copied
* update swagger docs
* update admin processer to use non-interface storage driver
* remove shortcode updating for local emojis
* go fmt
Co-authored-by: f0x52 <f0x@cthu.lu>
* [bugfix] Prevent future statuses entering timeline
Statuses created more than 5 minutes into the future are now rejected in the visibility package.
* Come on buddy
* [feature/performance] Fail fast when doing remote transport calls inside incoming request contexts
* [chore] Reduce outgoing request timeout to 15s
* log error messages when fastfailing
* use context.Value() instead of wrapped context, wrap error with fastfail instead of extra log entry
* add fast-fail context key test
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* replace s3 storage implementation to also use kv.KVStore
Signed-off-by: kim <grufwub@gmail.com>
* pull in latest `go-store` fix
Signed-off-by: kim <grufwub@gmail.com>
* pull-in go-store v2.0.9 fixes, update s3 put chunk size to 5MiB
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* [enhancement] markdown: disable SmartyPantsFractions
fixes#1028
* some fractions are still converted to unicode
to fix that, we might need to disable smartypants indeed in its entirety
* disable smartypants completely
for lack of a better simple solution
* add account block DB cache and remove reliance on relational joins
* actually include cache key arguments...
* add a PutBlock() method which also updates the block cache, update tests accordingly
* use `PutBlock` instead of `Put(ctx, block)`
* add + use functions for deleting + invalidating blocks
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* refactor federator account statuses delete to better catch errors, ensure next maxID is always set
Signed-off-by: kim <grufwub@gmail.com>
* fix error statement missing 2nd format operator
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
Google abandoned Floc in favour of Topics and changed the
necessary permissions policy. Currently, the Floc policy will
block Topics. This change includes switching to the updated
policy to be ahead of Google abandoning recognising the Floc
policy.
Update the function documentation to include the current relevant
documentation.
* convert most of the caches to use result.Cache{}
* add caching of emojis
* fix issues causing failing tests
* update go-cache/v2 instances with v3
* fix getnotification
* add a note about the left-in StatusCreate comment
* update EmojiCategory db access to use new result.Cache{}
* fix possible panic in getstatusparents
* further proof that kim is not stinky
* [bugfix] Fix unicode-unaware word boundary check in hashtag regex
Go `\b` does not care for Unicode, and without lookahead, the workarounds got
very ugly. So I replaced the regex with a parser.
The parser runs in O(n) time and performance should not be affected.
* [bugfix] Add back hashtag max length and add tests for it
* [feature] Add 'role' field to api serialization of local accounts
* [chore] Add a bit of license text while I'm here
* [frogend] render account role on same line as username in web view of profile
* style tweaking on role badges, general profile header layout
* profile stats wrapping
* don't render standard 'user' role on web view
Co-authored-by: f0x <f0x@cthu.lu>
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API
* [feature] Create (or use existing) category for new emoji uploads
* fix lint issue
* update misleading line in swagger docs
* Add instance-expose-public-timeline flag
Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.
* Update structure following review
* Add comment
* Fix linting
* Fix login on Mastodon iOS app for users with no statuses
Mastodon for iOS can't cope with an empty string for a date and
expect a JSON `null` instead.
Fixes https://github.com/superseriousbusiness/gotosocial/issues/1010
* Fix expected values in tests to match
* fix incorrect static remote url use for emojis
* warn when emoji/attachment already exists
* defer emoji postdata execution
* rename ctx to innerCtx for clarity
* warn on emoji too large
* small efficiency fix in fetchRemoteAccountEmojis
* tidy up lock+load
* lock processing emojis
* fix little fucky wucky
* this wasn't go fmted for some reason
* [feature] Read + Write tombstones for deleted Actors
* copyTombstone
* update to use resultcache instead of old ttl cache
Signed-off-by: kim <grufwub@gmail.com>
* update go-cache library to fix result cache capacity / ordering bugs
Signed-off-by: kim <grufwub@gmail.com>
* bump go-cache/v3 to v3.1.6 to fix bugs
Signed-off-by: kim <grufwub@gmail.com>
* switch on status code
* better explain ErrGone reasoning
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* S3: add config value "proxy" for not redirecting
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
* S3: document new config value "proxy"
* S3: add new config value "proxy" to test scripts
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
* [feature] Make instance thumbnail configurable via admin panel
* log db errors in InstanceToAPIInstance
* only update instance in db if necessary
* start adding tests
* finish test
* re-add eslint
* fix oauth url getting too long
* actually attach single emoji get and delete routes
* basic emoji details + deletion using rtk query
* refactor emoji upload to rtk query
* clean up old redux api+reducers for custom emoji
* fix validation order
* refactor custom emoji form fields
* remove unused requires
* cleanup, fix most eslint errors
* more small eslint fixes
* fix max emoji size
* tiny bit of function documentation
* greatly simplify httpclient request queuing
Signed-off-by: kim <grufwub@gmail.com>
* improved request queue mutex logic
Signed-off-by: kim <grufwub@gmail.com>
* use improved hashmap library
Signed-off-by: kim <grufwub@gmail.com>
* add warn logging when request queues are full
Signed-off-by: kim <grufwub@gmail.com>
* improve worker pool prefix var naming
Signed-off-by: kim <grufwub@gmail.com>
* improved worker pool error logging
Signed-off-by: kim <grufwub@gmail.com>
* move error message into separate field
Signed-off-by: kim <grufwub@gmail.com>
* remove old log statement
Signed-off-by: kim <grufwub@gmail.com>
* don't export worker message, it gets very spammy :')
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* bump go-store version to v2.0.5, init kv.KVStore without initial clean (as we are using for storage, not as a key-value store)
Signed-off-by: kim <grufwub@gmail.com>
* remove newline
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
If set, the landing page user configuration value is used as a Gin
context parameter, which seems incorrect, since a normal request isn't
going to have a parameter named after an arbitrarily configured user.
Instead, the user name should be used directly when building the
redirect URL.
* use readcloser for content.Content
* call media postdata function no matter what
* return a readcloser from data func
* tidy of logic of readertostore
* fix whoopsie
