mirrors/pict-rs
1
0
Fork 0
mirror of https://git.asonix.dog/asonix/pict-rs.git synced 2024-05-28 16:38:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
1362 commits 20 branches 305 tags 17 MiB
Commit graph

631 commits

Author SHA1 Message Date
asonix 1b97ac1c5a Start replacing manual stream implementations with streem 2023-09-10 18:55:13 -04:00
asonix 858899b943 Add metrics & tracing to dropped uploads, turn pop into a single query 2023-09-07 19:20:41 -05:00
asonix 783df60f81 Fix on_conflict for postgres store migration path 2023-09-06 20:49:00 -05:00
asonix a761daca34 clippy 2023-09-05 20:55:11 -05:00
asonix e15a82c0c7 Add external validation check 2023-09-05 20:45:07 -05:00
asonix bc49f8ca37 Migrate with timestamp 2023-09-04 22:25:11 -05:00
asonix 9c50bbe23a match -> matches! 2023-09-04 22:06:28 -05:00
asonix 492b99922b 10 second worker delay after disconnect error 2023-09-04 22:04:15 -05:00
asonix 62ebc1c60a Move timeout after metrics - timeouts count as incomplete 2023-09-04 22:02:59 -05:00
asonix 3bd0f78e75 Use with_timeout for all timeouts 2023-09-04 21:58:57 -05:00
asonix a4b1ab7dfb Instrument postgres db calls 2023-09-04 21:51:27 -05:00
asonix 37448722de postgres: Rework job & upload notifications (more) 
postgres: Add metrics to job push & pop, upload wait
sled: add upload wait metrics
 2023-09-04 16:20:26 -05:00
asonix 00aa00c55c postgres: Enable 'previous page' when less than full limit fit on previous page 2023-09-03 21:58:31 -05:00
asonix a43de122f9 postgres: add already-claimed case, general: tracing paranoia 2023-09-03 21:30:47 -05:00
asonix 31caea438e Fix slow connection pool access 2023-09-03 20:05:29 -05:00
asonix 8a892ba622 it might work 2023-09-03 18:21:46 -05:00
asonix a2933dbebc Implement all the todos 2023-09-03 17:11:34 -05:00
asonix 94cb2a9ef3 Pass clippy 2023-09-03 16:59:41 -05:00
asonix 443d327edf Implement a couple more repo traits 2023-09-03 12:47:06 -05:00
asonix eac4cd54a4 Initial work for pg notifications 2023-09-02 20:13:32 -05:00
asonix e580e7701e Finish implementing HashRepo 2023-09-02 19:13:00 -05:00
asonix 8921f57a21 Remove Identifier trait, Replace with Arc<str> 2023-09-02 18:38:02 -05:00
asonix 8eb4cda256 Implement first query 2023-09-02 13:52:15 -05:00
asonix d475723087 Finish writing migrations & generate diesel schema 2023-09-02 13:35:30 -05:00
asonix 8c532c97e6 Initial postgres work 2023-09-02 11:52:55 -05:00
asonix ba3a23ed43 Add error codes 2023-09-01 20:51:01 -05:00
asonix fd74161c61 Rename some repo methods, generate UploadId in repo 2023-09-01 18:41:04 -05:00
asonix cbb66f1b75 Add internal endpoint for deleting alias without a delete token 2023-09-01 17:25:13 -05:00
asonix 8fb90a6f69 Give a meaningful distinction for format in details 2023-08-31 16:26:45 -05:00
asonix dce0827099 Increase default animation bounds, remove references to converting animations to videos 2023-08-30 21:55:53 -05:00
asonix 0b70ca49e4 clippy 2023-08-30 21:00:15 -05:00
asonix 721d9e1706 Don't try parsing an internal format for OldDetails 2023-08-30 20:55:38 -05:00
asonix 92ee60549f Enable copying input av1 stream from mp4 to webm 2023-08-30 20:40:08 -05:00
asonix b48a9233b2 Remove transcode from animation to video, make video transcoding 'optional' 
Video transcoding still happens, but in many cases the video stream is able to be copied verbatim rather than being decoded & encoded
 2023-08-30 20:37:54 -05:00
asonix 08fd96c2f7 clippy nit 2023-08-29 13:29:40 -05:00
asonix c0f6b1e988 Enable searching hashes by date 2023-08-29 13:27:18 -05:00
asonix 0146202236 Fully commit to HumanDate 2023-08-29 12:59:36 -05:00
asonix 64950bfe0e Shorten pagination slugs 2023-08-29 12:53:29 -05:00
asonix 1271ff2cc7 Clippy 2023-08-28 18:49:30 -05:00
asonix 2b6ea97773 Enable setting timestamp on hash creation, improve paging api 2023-08-28 18:43:24 -05:00
asonix 24812afeba Add paged hash access 2023-08-28 16:02:11 -05:00
asonix e302ab1f3d Migrate not-found alias 2023-08-28 13:40:19 -05:00
asonix b28cc38c38 Simplify Streamer 2023-08-23 19:12:59 -05:00
asonix 95637fdfe5 Remove direct dep on futures-util 2023-08-23 19:10:10 -05:00
asonix 8f50a15b25 Reduce reliance on futures-util 2023-08-23 11:59:42 -05:00
asonix 0352ada9d8 Don't doulbe-open repo 2023-08-16 16:39:41 -05:00
asonix 88d4cc1a6a Exit after repo migration 2023-08-16 16:34:42 -05:00
asonix f95e61e402 Expose repo migrations 2023-08-16 16:32:19 -05:00
asonix ba27a1a223 Build out repo-repo migration 2023-08-16 16:09:40 -05:00
asonix 1559d57f0a Don't overwrite existing variants 2023-08-16 15:12:16 -05:00
asonix 22cfbe979d Note how many failures have occurred for a given hash 2023-08-16 13:29:22 -05:00
asonix ee88823d41 clippy 2023-08-16 12:43:12 -05:00
asonix 521cdd5b9d Enforce format on new details 2023-08-16 12:36:18 -05:00
asonix f3c6239979 Allow skipping details generation if already generated 2023-08-16 11:57:09 -05:00
asonix 4d909ba32a Make migration concurrent 2023-08-16 11:47:36 -05:00
asonix ac996ff832 Remove unused error 2023-08-15 21:25:04 -05:00
asonix 1de257bb07 Ensure access values are unique 2023-08-15 21:18:25 -05:00
asonix bd3b668d81 Don't error on duplicate - happens on retried migration 2023-08-15 20:28:42 -05:00
asonix 9fe7410dd6 Add logging to migration 2023-08-15 19:33:19 -05:00
asonix 1ee77b19d8 Make Repo object safe. Create migration 2023-08-15 19:19:03 -05:00
asonix 09236d731d Fix poll after completion, misused compare_and_swap 2023-08-14 21:17:57 -05:00
asonix 26ca3a7195 Remove associated Bytes type 2023-08-14 20:00:00 -05:00
asonix da876fd553 Make it compile 2023-08-14 14:25:19 -05:00
asonix 3129f7844e BROKEN: start work on hash discriminant 2023-08-13 22:06:42 -05:00
asonix e4e93cddf9 Return JobId from push 2023-08-13 19:52:27 -05:00
asonix 5d4486fdf4 Finish queue implementation update for sled 2023-08-13 19:47:20 -05:00
asonix 35d0c065e9 Start work on 0.5 db format - initial queuerepo changes 2023-08-13 14:12:38 -05:00
asonix 0c4d632dda Consider timeouts a client error 2023-08-05 16:18:06 -05:00
asonix fee4ed1e3e Add process timeout 2023-08-05 12:41:06 -05:00
asonix 5f12be0c6a Clippy 2023-07-26 22:53:41 -05:00
asonix 2a59a6492a Bring back hash_alias_key 2023-07-25 20:30:22 -05:00
asonix bd3975f455 Simplify some repo functions, remove 0.3 migration 2023-07-25 20:08:18 -05:00
asonix f221887599 Clippy 2023-07-23 16:04:17 -05:00
asonix aac1bb7bc4 Finish media proxy implementation 2023-07-23 15:45:52 -05:00
asonix 4cca7d0f86 Prepare for proxying media 2023-07-23 12:10:36 -05:00
asonix 75bf425c6e Update alias query, process query to accept proxy url 2023-07-23 11:45:10 -05:00
asonix e59483c12c Add ProxyRepo 2023-07-23 10:23:46 -05:00
asonix d0c619bdf7 Clippy 2023-07-22 21:14:24 -05:00
asonix 77cdeab57e Improve metrics, add job metrics 2023-07-22 21:11:28 -05:00
asonix 7dc8ede8eb Gate cleanup behind read-only config 2023-07-22 20:26:28 -05:00
asonix 13d6e10886 Reduce log level on variants access 2023-07-22 20:20:58 -05:00
asonix 1f3298363f Attempt to mark unmarked variants as accessed on launch 2023-07-22 19:55:50 -05:00
asonix fe1f7c869f Enable cleanup of variants 
Still TODO: doing a first pass to mark variants as accessed on launch
 2023-07-22 19:41:50 -05:00
asonix 47e13ec04e Mark variants as accessed on fetch 2023-07-22 18:50:04 -05:00
asonix b9e6d67d15 Add retention configuration 
This doesn't do anything yet, but is part of the Media Proxy and Variant Cleanup tickets
 2023-07-22 18:37:25 -05:00
asonix 13387dec43 Fix specs 2023-07-22 17:58:03 -05:00
asonix b786406ad0 Change IdentifierAccess to VariantAccess 2023-07-22 17:57:52 -05:00
asonix ce0df080f4 Add prometheus metrics 2023-07-22 16:47:59 -05:00
asonix d661732fb4 Tidy imports 2023-07-22 12:47:32 -05:00
asonix a390b21df1 Make running pict-rs fluent 2023-07-22 12:39:39 -05:00
asonix 3ca994ee31 Replace global config with passed-around config 
This will enable spinning up pict-rs multiple times in the same process with different configurations
 2023-07-22 12:31:01 -05:00
asonix e1262a5fda Extract preprocess steps as urlencoded in config deserialization 2023-07-22 11:38:09 -05:00
asonix ac48003f45 Extract ProcessMap out of static 2023-07-22 11:16:19 -05:00
asonix 8d35c2449d Simplify the cancel-safe processor 2023-07-22 10:46:16 -05:00
asonix c1127b4117 Set content-length in PutObject, CompleteMultipart 2023-07-21 19:45:22 -05:00
asonix 2f0a3618d8 Replace awc with reqwest 2023-07-21 16:58:31 -05:00
asonix fc14ac2038 Clippy 2023-07-18 21:59:27 -05:00
asonix 3d93300ba9 Add access repos for alias & identifier 2023-07-18 21:56:13 -05:00
asonix 127dd6cdad Make max file count configurable 2023-07-18 20:32:17 -05:00
asonix eeac900d7e Add quality value to process method, move -coalesce earlier in pipelines 2023-07-18 16:18:01 -05:00
asonix 558605381d Fix dangling unprocessed uploads 
Adds error boundary around backgrounded ingest
 2023-07-17 22:32:08 -05:00
asonix 496bcc5dc1 Only include -b:v 0 for VP9 
also simplify building arguments
 2023-07-17 18:13:50 -05:00
asonix 3aecef3a9e Remove println 2023-07-17 17:47:48 -05:00
asonix b1bbc6b159 Attempt adding quality settings to pict-rs 2023-07-17 17:45:26 -05:00
asonix c4c920191f Add read-only mode 2023-07-17 14:24:49 -05:00
asonix dd1d509bb1 Enable configuring download client timeout 2023-07-17 13:44:31 -05:00
asonix 5e8ab7856d Extract Status errors into command-specific errors 2023-07-17 13:30:08 -05:00
asonix 2961aae6e3 Port migration changes from 0.4.1 2023-07-16 22:07:42 -05:00
asonix a751d92436 Port error changes to 0.5 2023-07-16 21:51:14 -05:00
asonix 9871ad2ee0 Clippy 2023-07-14 23:16:57 -05:00
asonix 4d78ce7689 Fix store delegation for public_url 2023-07-14 15:23:07 -05:00
asonix 786f583d98 Add public_url to Store for optional redirects to public object store urls 
Add details generation to file migration to set the content-type for uploads
 2023-07-14 14:53:37 -05:00
asonix 4e58388b41 Clean alias before checking hash option 2023-07-13 19:58:31 -05:00
asonix afeac8294a Fix Alias cleanup ordering 
Update dev.toml
 2023-07-13 19:57:09 -05:00
asonix ec6c9aac1e Better discovery validation, proper invoking of imagemagick 2023-07-13 19:21:57 -05:00
asonix 9b1dd1f8c3 Copy process from 0.4.x branch 2023-07-13 19:21:28 -05:00
asonix 2c22f2ee3a It compiles and runs, but doesn't work 2023-07-13 17:42:21 -05:00
asonix 27451971a6 Fix tests, add animated_webp test 2023-07-13 14:34:40 -05:00
asonix 33b83f97f2 It compiles 
Need to reinstate media limits
 2023-07-13 13:48:59 -05:00
asonix ad1837f9dd VERY BROKEN: start replacing parts of pict-rs 2023-07-12 22:12:21 -05:00
asonix 58d9765594 WIP: Rework format detection and conversions 
- Add a bunch of types & methods that aren't used yet. The idea is
  creating the output from the known input parameters is pure, and works
  for any uploaded media
- Introduce a new kind of media "animation" which is neither an image
  nor a video. It is likely that animations can be processed properly at
  the /image/process.{ext} endpoint with a little massaging

TODO:
- Integrate this into exiftool, ffmpeg, magick commands
- Detect apng with ffmpeg (imagemagick treats it as a still png unless it's given a
    hint)
- Infallible conversion from Details to OutputFile - this might be
  impossible given that we don't currently store more than a mime type
  and a frame count to give any sort of indication of real type
- Try not to break API
 2023-07-11 23:11:25 -05:00
asonix 691bca286c Make object storage timeouts configurable 2023-07-11 13:01:58 -05:00
asonix f5c39f9be5 Typo, error formatting 2023-07-10 17:15:43 -05:00
asonix 4be2eb67d1 Increase object storage signature expiry to 15 seconds 2023-07-10 16:28:51 -05:00
asonix 9004ecaadf Better classify process related errors 2023-07-10 15:29:41 -05:00
asonix e10cb1401d Bump version 2023-07-09 15:16:04 -05:00
asonix 7dadbcf3ed Replace UnsupportedFormat error with more helpful errors 2023-07-09 15:07:49 -05:00
asonix 1694f49436 Use json for ffmpeg, imagemagick details parsing 2023-07-09 14:50:58 -05:00
asonix 686be1d93b Turn export_path into flag for sled 2023-07-08 19:56:26 -05:00
asonix 7768d4e58e Add database exports 
This should help admins running pict-rs on traditional filesystems to
keep valid backups of pict-rs' sled repo
 2023-07-08 17:36:01 -05:00
asonix 4680f9c3aa Instrument repo and store build methods 2023-07-08 10:42:03 -05:00
asonix d51c348a3f Run health checks on old and new stores before migrating 2023-07-07 14:05:13 -05:00
asonix 3ccb8ecd8c Validate alias represents real file before setting not found 2023-07-07 13:33:27 -05:00
asonix 9e7376d411 Remove is_missing error, make Identifier and DeleteToken return Option on fetch 2023-07-07 13:17:26 -05:00
asonix 7274538b70 Try adding store healthcheck 2023-07-07 12:05:42 -05:00
asonix c150903980 Expand store differentiation 2023-07-07 11:19:39 -05:00
asonix 75be876d3c Differentiate missing files in old vs new store for migration 2023-07-07 11:03:10 -05:00
asonix 508dd4340b Fix panic when migrating < 100 files, improve resumed migration messaging 2023-07-06 16:06:37 -05:00
asonix 3487cb0e30 Add ability to set 404 image 
Fix imagemagick blur arguments
 2023-07-05 19:07:03 -05:00
asonix e7e4876908 Explicitly handle cases of invalid aliases 2023-07-05 16:46:44 -05:00
asonix 23e67b9697 Enhance migration handling of missing identifiers 
Queue a cleanup task for hashes that don't have original file identifiers
Be more specific about what field is missing when a field is missing
 2023-07-05 09:52:19 -05:00
asonix 2274dfecb4 Fix io-uring - read_to_end was polling Stream mapped through StreamReader after completion 2023-07-05 00:35:07 -05:00
asonix c8e84b8914 Greatly improve error information during migration 2023-07-02 21:07:15 -05:00
asonix e6f17cbac6 Log on successful migration 2023-06-29 11:45:06 -05:00
asonix 01d3610f46 Add basic progress indication for store migration 2023-06-29 11:39:47 -05:00
asonix 0d1df5ebc2 Avoid failed identifier cleanup on failed duplicate upload 2023-06-23 11:43:15 -05:00
asonix 7ba2fc9b26 Fix duplicate uploads potentially purging existing uploads on failure 2023-06-23 11:39:43 -05:00
asonix 58f0c328d9 Fix typo in error message 2023-06-23 11:20:55 -05:00
asonix a1d48cbb42 Share client between object store and request extractor 
Add config to limit connection pool size
 2023-06-23 11:20:20 -05:00
asonix 071d788349 Include original error in object storage sendrequest error 2023-06-21 17:07:14 -05:00
asonix a3ff072350 Remove unused cache duration parameter 2023-06-21 17:05:53 -05:00
asonix 6d2aef8cc0 Add jpegxl and avif support 2023-06-21 17:05:35 -05:00
asonix 554d852e68 Create unique errors for Repo and Store, separate from UploadError 
Add .is_not_found() for Store errors, wire up Object and File storage to properly set NotFound
Allow skipping files that are not found during store migration
 2023-06-20 15:59:08 -05:00
asonix 1214b51ad7 Make use of FilesystemDefaults when constructing migrate commands 2023-06-19 15:04:36 -05:00
asonix a3a986638d Improve gif handling. 
- Allow a fast-path exif cleaning if we'd re-encode a gif to a gif
- Use single-frame palettes to better map transparency from frame to frame

Unrelated:
- decrease ffmpeg logging when creating thumbnails
 2023-06-19 14:25:31 -05:00
asonix 22e526fd2a Allow type underscore (probably in instrument) 2023-06-01 17:33:43 -05:00
asonix cc314c37a6 Use libopus for ffmpeg 2023-03-09 21:29:56 -06:00
asonix dc27200a35 Add transparency to gifs, rustfmt 2023-02-25 14:31:57 -06:00
asonix 97ba4e806b Clippy 2023-02-25 13:01:55 -06:00
asonix 9a2f846cd1 Detect input transparency for gif/webm 2023-02-25 12:59:47 -06:00
asonix 281ac43dff Remove cache functionality 2023-02-25 11:34:48 -06:00
asonix 123488f617 clippy 2023-02-12 23:18:27 -06:00
asonix 788b87eceb Remove obsolute pre-0.3 migrate check 2023-02-12 23:13:47 -06:00
asonix 138499d47f Use transpose in a couple places 2023-02-05 20:50:59 -06:00
asonix cc75d2d882 Clippy 2023-02-04 17:54:56 -06:00
asonix 4cc810d372 Add control over gif frame count 2023-02-04 17:52:23 -06:00
asonix 88ca9793e8 s/vrames/frames:v 2023-02-04 17:47:03 -06:00
asonix 40f57be0c7 Allow uploading small gifs 2023-02-04 17:32:36 -06:00
asonix 0aa3f574a5 Clippy 2023-01-29 11:57:59 -06:00
asonix f6d6d54b88 Update deps 2023-01-29 11:47:28 -06:00
asonix f0f40db8c3 Add healthcheck for db 2023-01-29 11:36:09 -06:00
asonix 5e5dd27a05 Retry migrate_file, remove variant before relating new one 2023-01-04 18:58:05 -06:00
asonix 28f7a139a0 Retry in-process, and rate-limit self 2023-01-04 17:26:05 -06:00
asonix c55b1115d1 Increase client timeout, attempt to keep track of migration progress for resuming 2023-01-04 15:58:32 -06:00
asonix dabcc24f98 Parse environment variables - main app 2023-01-04 15:05:34 -06:00
asonix a489411c6c Parse environment variables 2023-01-04 14:42:04 -06:00
asonix 3c844d86c5 Allow configuring pict-rs through serializable types 2022-12-26 16:35:25 -06:00
asonix 2c1a37fb62 Extract pict-rs as library 2022-12-23 12:56:15 -06:00
asonix 74cd3392fa Borrow parent spans when creating children 2022-12-07 22:43:12 -06:00
asonix 3325b222a2 clippy 2022-11-26 11:18:34 -06:00
asonix 7037ee5791 Support auto-orienting photos 2022-10-15 11:13:24 -05:00
asonix 669b3fb86f Fix webp metadata stripping, more tracing cleanup 2022-10-01 22:47:52 -05:00
asonix 718f09c43a Clean tracing, simplify validation, rename InputFormat -> VideoFormat 2022-10-01 21:17:18 -05:00
asonix e7cf21f862 Allow concurrency in small object uploads 2022-10-01 13:06:19 -05:00
asonix fe2a499110 Fix sending upload parts (set size), optimize small file uploads 2022-10-01 13:00:07 -05:00
asonix 0307dc5a3b Introduce BytesStream to simplify aggregating, forwarding bytes 2022-10-01 12:06:33 -05:00
asonix c767804430 Default to vp9 for uploaded video 2022-10-01 10:17:32 -05:00
asonix 85c5ce955d Clippy 2022-09-30 20:02:46 -05:00
asonix d86d9a3228 Put proper file extension on videos 2022-09-30 20:00:14 -05:00
asonix 8eb2293808 Enable selecting video and audio codecs for uploaded media 2022-09-30 19:38:11 -05:00
asonix f4542efcc1 Share ffprobe details code 2022-09-30 17:43:40 -05:00
asonix 3ceaaa391c Update clap 2022-09-28 18:23:41 -05:00
asonix 656b0225af Clippy 2022-09-27 23:23:06 -05:00
asonix 29cab025c6 Improve 0.3 migration code, repo traces 2022-09-27 23:19:52 -05:00
asonix 85f20bbe0e Add new resize options, update README with current APIs 2022-09-26 18:14:25 -05:00
asonix 9db057fdc5 Add admin endpoint to fetch identifier from an alias 2022-09-25 20:59:57 -05:00
asonix 17dab63662 Clippy 2022-09-25 20:44:24 -05:00
asonix 80c83eb491 Add ffprobe for details inspection - vastly improve video detection speed 2022-09-25 20:39:09 -05:00
asonix 5449bb82f1 Better identify video uploads with ffprobe 2022-09-25 19:34:51 -05:00
asonix 21e3e63ac3 Instrument upload store methods 2022-09-25 18:46:50 -05:00
asonix 51d8621015 Defensively format errors in traces to avoid deadlocks 2022-09-25 18:46:26 -05:00
asonix 890478e794 Support audio in uploaded videos, allow webm uploads 2022-09-25 18:16:37 -05:00
asonix c57a48db8a Start introducing more video controls 2022-09-25 17:36:07 -05:00
asonix c9a74a73ca Reduce redundancy in process module 2022-09-25 17:35:52 -05:00
asonix 50d118a2a7 Implement image pre-processing on upload 2022-09-25 15:17:33 -05:00
asonix 04bc586a26 Fix hasher test 2022-09-25 09:09:05 -05:00
asonix dc95417846 Improve concurrency for upload streams part 2 2022-09-25 08:27:11 -05:00
asonix bf3c47e457 Improve concurrency for upload streams 2022-09-24 22:07:06 -05:00
asonix a6adde874e Implement s3 multipart uploads 2022-09-24 20:33:59 -05:00
asonix 25209e29c0 Make it compile 2022-09-24 17:18:53 -05:00
asonix ff1771e016 More migration work for rusty-s3 2022-09-24 14:18:49 -05:00
asonix ab7fd9aaf7 Start work on using rusty-s3 instead of rust-s3 2022-09-24 13:39:27 -05:00
asonix c5b107eb4c clippy 2022-09-11 10:04:37 -05:00
asonix 7f49af58fe Update actix-form-data, impl Store & Repo for Data<T> 2022-09-10 19:29:22 -05:00
asonix e0bc95f785 Remove unneeded reborrow 2022-08-15 15:00:12 -05:00
Aode (lion) 547934071a Don't migrate from the old database if it doesn't exist 2022-08-14 11:49:20 -05:00
Aode (lion) e0543b2c1c Add HEAD endpoints for process, serve 2022-07-17 11:15:43 -05:00
asonix cb9a594f24 Fix content type on already processed images 2022-06-21 11:27:22 -05:00
asonix dc73bdbad4 Make already-claimed a 400 2022-06-05 23:24:58 -05:00
Aode (lion) 2db76ad8c5 Update dependencies 2022-04-29 15:54:54 -05:00
Aode (lion) 8ee26ccf4f change some import paths 2022-04-27 20:13:49 -05:00
Aode (Lion) 4ac20546ce Improve migration from 0.3 formats 2022-04-24 17:10:15 -05:00
Aode (Lion) b3cbda1337 Add internal endpoint for cleaning variants 2022-04-11 16:56:39 -05:00
Aode (Lion) e50ccfb48c Update deps 2022-04-11 15:17:54 -05:00
Aode (lion) 05533f7e3a Use rfc3339 dates for details responses 2022-04-08 13:36:06 -05:00
Aode (lion) 7436f15267 Dry download client logic 2022-04-08 13:16:22 -05:00
Aode (lion) 5f19b63166 Extract details processing 2022-04-08 13:03:00 -05:00
Aode (lion) 31c5a36c77 Better instrument drops, jobs. Properly disarm backgrounded downloads 2022-04-08 12:51:33 -05:00
Aode (lion) 55d5e43cd2 Consolidate endpoints for downloading media, cached or backgrounded 2022-04-08 12:05:14 -05:00
Aode (lion) 07c61cbe87 Add download backgrounded 2022-04-07 19:07:30 -05:00
Aode (lion) 8781bc8f28 Add follows-from relation for background tasks 2022-04-07 13:28:28 -05:00
Aode (lion) e493e90dd4 Defensive tracing 2022-04-07 12:56:40 -05:00
Aode (lion) c80d207a87 Improve tracing 2022-04-06 21:40:49 -05:00
Aode (Lion) 05e2cf5e08 Cache on Alias rather than Hash 
Make sled repo blocking tasks share parent span
 2022-04-06 12:14:12 -05:00
Aode (Lion) 865f4ee09c Caching works: 
- Insert new bucket for current access time before removing
    previous access time record.
- Add instrument to more methods for more precise tracing
 2022-04-05 21:47:35 -05:00
Aode (Lion) 6cdae7b318 Add 'cache' endpoint for ingesting ephemeral media 
By default, cached media should only stick around for 7 days, however
The timeout is reset every time media is accessed, so only obscure
cached media will be flushed from the cache. This '7 days' number is
configurable through the commandline run options as --media-cache-duration
and in the pict-rs.toml file as [media] cache_duration
 2022-04-05 20:29:30 -05:00
Aode (lion) b0670c2f66 Move migrate to old 2022-04-03 15:07:31 -05:00
Aode (lion) 409b7d4c54 Remove unused conversions 2022-04-03 12:51:21 -05:00
Aode (lion) 29f0774331 Add ability to claim uploads 2022-04-02 21:15:39 -05:00
Aode (lion) 8734dfbdc7 Add backgrounding routes 
- Accept backgrounded uploads
- Allow backgrounded processing

Still TODO:
- Endpoint for waiting on/claiming an upload
 2022-04-02 20:56:29 -05:00
Aode (Lion) c4d014597e Better job recovery 2022-04-02 18:53:03 -05:00
Aode (Lion) 77a400c7ca Implement UploadRepo 2022-04-02 17:41:00 -05:00
Aode (Lion) 132e395e5c Multiple items: 
- Reduce duplicate work in generate job
- Use hash instead of identifier for unique processing
- Move motion ident generation behind concurrent processor lock
 2022-04-02 17:40:08 -05:00
Aode (Lion) 09f53b9ce6 Move away from UploadManager to direct repo & store actions 2022-04-02 16:44:03 -05:00
Aode (lion) 6ed592c432 In Progress: process jobs 
- Is this a good idea? it definitely will make interacting with pict-rs harder. Maybe it's best not to do this
 2022-04-01 16:51:12 -05:00
Aode (lion) c0d8e0e8e3 Support multiple queues for job processor 2022-04-01 11:51:46 -05:00
Aode (lion) 09281d9ae8 Move stream timeout to all response bodies 2022-03-29 16:48:26 -05:00
Aode (lion) 63d66050c8 Enable buffering iterator items for iterstream 2022-03-29 16:18:00 -05:00
Aode (lion) 5adb3fde89 Move some Stream implementations into 'stream' module 2022-03-29 15:59:17 -05:00
Aode (lion) 0e490ff54a Remove custom BytesFreezer stream 2022-03-29 14:28:08 -05:00