mirrors/pict-rs
1
0
Fork 0
mirror of https://git.asonix.dog/asonix/pict-rs.git synced 2024-12-11 02:36:30 +00:00
Code Issues Projects Releases Packages Wiki Activity
1350 commits 27 branches 315 tags 6.2 MiB
Commit graph

624 commits

Author SHA1 Message Date
asonix
71fbc4c55b Just have different mains 2023-10-07 12:31:05 -05:00
asonix
760b9d8c59 Install tracing & metrics from within runtime 2023-10-07 12:29:01 -05:00
asonix
3f5bc1d311 Build same pict-rs config for either runtime option 2023-10-07 12:22:17 -05:00
asonix
f97f76c01e Fix io-uring initialization 2023-10-07 12:12:40 -05:00
asonix
4a45aa889c Enable running on bare tokio 2023-10-07 11:48:58 -05:00
asonix
7f5cbc4571 Attempt clean-drop of tmp_dir 2023-10-07 11:36:49 -05:00
asonix
7b5a3020fa Don't rely on global state for tmp_dir 2023-10-07 11:32:36 -05:00
asonix
fff4afe105 Start working on per-server temporary directory 2023-10-06 19:42:24 -05:00
asonix
c202484e9d clippy 2023-10-04 12:46:38 -05:00
asonix
914e21c043 Implement constant-time equality for delete tokens, inline alias cleanup 2023-10-04 12:11:29 -05:00
asonix
d28c1ac628 Upgrade Concurrency: Update docs, configs, limit maxiumum details operations 2023-10-03 17:04:40 -05:00
asonix
f223d477c7 Enable specifying concurrency for 0.4 to 0.5 migration 2023-10-03 16:27:19 -05:00
asonix
75551dc3a1 Increase deadpool wait timeout, helps with extreme concurrency 2023-10-03 16:15:49 -05:00
asonix
247902e600 Enable specifying migrate-store concurrency 2023-10-03 15:50:53 -05:00
asonix
8568ae3364 Clippy 2023-09-30 22:47:49 -05:00
asonix
4c7067d4ca Allow running without old_repo specified 2023-09-30 22:46:15 -05:00
asonix
81c6e73b5f Simplify old_repo, remove client_pool_size 2023-09-30 22:46:15 -05:00
asonix
47751f3875 Downgrade some WARN logs to INFO, update docs for public methods 2023-09-30 17:52:58 -05:00
asonix
01cbe34ac3 Remove unneeded annotation 2023-09-30 17:34:05 -05:00
asonix
4bb9d81f51 Move internal middleware into own file 2023-09-30 17:33:01 -05:00
asonix
e3462f6664 payload middleware: switch to Rc, always inject if Payload isn't None 2023-09-30 17:24:48 -05:00
asonix
3267fb8301 Fail requests for malformed deadlines 2023-09-30 17:20:32 -05:00
asonix
65ea518542 Default log level to info 2023-09-30 16:28:04 -05:00
asonix
66e1711723 Enable proper draining of dropped request payloads 
Doing this as the outermost middleware ensures all endpoints are covered.

Update request deadline to turn negative deadlines into immediate failures
 2023-09-30 16:26:43 -05:00
asonix
6f44b05c0a Remove references to Arbiter 2023-09-29 11:54:43 -05:00
asonix
735fcd1b49 Remove dependency on actix-web-codegen 2023-09-24 15:52:31 -05:00
asonix
ecdbaa9c92 Use diesel-async re-export of Hook 2023-09-24 15:48:19 -05:00
asonix
745e990fa0 Remove direct dependency on actix-rt and actix-server 2023-09-24 15:45:10 -05:00
asonix
0926739d3c Remove once_cell 2023-09-24 15:32:00 -05:00
asonix
e0968dcce3 Remove serde-cbor, num_cpus 2023-09-24 15:22:57 -05:00
asonix
fd965bb1a5 Allow processing still images into animation formats 2023-09-24 13:20:33 -05:00
asonix
f3ef2d870b Generate details during ingest 2023-09-24 12:58:16 -05:00
asonix
8f62b0c325 clippy 2023-09-24 12:42:23 -05:00
asonix
9ffdd0432f Infer video thumbnail format from image format 2023-09-24 12:37:43 -05:00
asonix
6c921817e1 Enable thumbnailing animations, use dynamic dispatch for a number of async readers 2023-09-24 11:54:16 -05:00
asonix
cde4a72203 Add metrics to object storage body streams 2023-09-20 19:46:10 -05:00
asonix
49d3037358 Add metrics to object storage requests 2023-09-20 19:39:03 -05:00
asonix
7df6f3906e Add timeout, metrics to external validation request 2023-09-20 19:29:19 -05:00
asonix
d0646dee25 Add config option for external validation timeout 2023-09-20 19:23:55 -05:00
asonix
4a5b9712aa Combine map_ok and map_err underlying impl 2023-09-10 23:51:07 -04:00
asonix
2b10c48619 Update io-uring to use streem for file bytes 2023-09-10 23:45:24 -04:00
asonix
ea75ca24b5 Add track_caller to sync methods, update streem 2023-09-10 20:43:51 -04:00
asonix
b2674f06d0 More streme 2023-09-10 20:08:01 -04:00
asonix
1b97ac1c5a Start replacing manual stream implementations with streem 2023-09-10 18:55:13 -04:00
asonix
858899b943 Add metrics & tracing to dropped uploads, turn pop into a single query 2023-09-07 19:20:41 -05:00
asonix
783df60f81 Fix on_conflict for postgres store migration path 2023-09-06 20:49:00 -05:00
asonix
a761daca34 clippy 2023-09-05 20:55:11 -05:00
asonix
e15a82c0c7 Add external validation check 2023-09-05 20:45:07 -05:00
asonix
bc49f8ca37 Migrate with timestamp 2023-09-04 22:25:11 -05:00
asonix
9c50bbe23a match -> matches! 2023-09-04 22:06:28 -05:00
asonix
492b99922b 10 second worker delay after disconnect error 2023-09-04 22:04:15 -05:00
asonix
62ebc1c60a Move timeout after metrics - timeouts count as incomplete 2023-09-04 22:02:59 -05:00
asonix
3bd0f78e75 Use with_timeout for all timeouts 2023-09-04 21:58:57 -05:00
asonix
a4b1ab7dfb Instrument postgres db calls 2023-09-04 21:51:27 -05:00
asonix
37448722de postgres: Rework job & upload notifications (more) 
postgres: Add metrics to job push & pop, upload wait
sled: add upload wait metrics
 2023-09-04 16:20:26 -05:00
asonix
00aa00c55c postgres: Enable 'previous page' when less than full limit fit on previous page 2023-09-03 21:58:31 -05:00
asonix
a43de122f9 postgres: add already-claimed case, general: tracing paranoia 2023-09-03 21:30:47 -05:00
asonix
31caea438e Fix slow connection pool access 2023-09-03 20:05:29 -05:00
asonix
8a892ba622 it might work 2023-09-03 18:21:46 -05:00
asonix
a2933dbebc Implement all the todos 2023-09-03 17:11:34 -05:00
asonix
94cb2a9ef3 Pass clippy 2023-09-03 16:59:41 -05:00
asonix
443d327edf Implement a couple more repo traits 2023-09-03 12:47:06 -05:00
asonix
eac4cd54a4 Initial work for pg notifications 2023-09-02 20:13:32 -05:00
asonix
e580e7701e Finish implementing HashRepo 2023-09-02 19:13:00 -05:00
asonix
8921f57a21 Remove Identifier trait, Replace with Arc<str> 2023-09-02 18:38:02 -05:00
asonix
8eb4cda256 Implement first query 2023-09-02 13:52:15 -05:00
asonix
d475723087 Finish writing migrations & generate diesel schema 2023-09-02 13:35:30 -05:00
asonix
8c532c97e6 Initial postgres work 2023-09-02 11:52:55 -05:00
asonix
ba3a23ed43 Add error codes 2023-09-01 20:51:01 -05:00
asonix
fd74161c61 Rename some repo methods, generate UploadId in repo 2023-09-01 18:41:04 -05:00
asonix
cbb66f1b75 Add internal endpoint for deleting alias without a delete token 2023-09-01 17:25:13 -05:00
asonix
8fb90a6f69 Give a meaningful distinction for format in details 2023-08-31 16:26:45 -05:00
asonix
dce0827099 Increase default animation bounds, remove references to converting animations to videos 2023-08-30 21:55:53 -05:00
asonix
0b70ca49e4 clippy 2023-08-30 21:00:15 -05:00
asonix
721d9e1706 Don't try parsing an internal format for OldDetails 2023-08-30 20:55:38 -05:00
asonix
92ee60549f Enable copying input av1 stream from mp4 to webm 2023-08-30 20:40:08 -05:00
asonix
b48a9233b2 Remove transcode from animation to video, make video transcoding 'optional' 
Video transcoding still happens, but in many cases the video stream is able to be copied verbatim rather than being decoded & encoded
 2023-08-30 20:37:54 -05:00
asonix
08fd96c2f7 clippy nit 2023-08-29 13:29:40 -05:00
asonix
c0f6b1e988 Enable searching hashes by date 2023-08-29 13:27:18 -05:00
asonix
0146202236 Fully commit to HumanDate 2023-08-29 12:59:36 -05:00
asonix
64950bfe0e Shorten pagination slugs 2023-08-29 12:53:29 -05:00
asonix
1271ff2cc7 Clippy 2023-08-28 18:49:30 -05:00
asonix
2b6ea97773 Enable setting timestamp on hash creation, improve paging api 2023-08-28 18:43:24 -05:00
asonix
24812afeba Add paged hash access 2023-08-28 16:02:11 -05:00
asonix
e302ab1f3d Migrate not-found alias 2023-08-28 13:40:19 -05:00
asonix
b28cc38c38 Simplify Streamer 2023-08-23 19:12:59 -05:00
asonix
95637fdfe5 Remove direct dep on futures-util 2023-08-23 19:10:10 -05:00
asonix
8f50a15b25 Reduce reliance on futures-util 2023-08-23 11:59:42 -05:00
asonix
0352ada9d8 Don't doulbe-open repo 2023-08-16 16:39:41 -05:00
asonix
88d4cc1a6a Exit after repo migration 2023-08-16 16:34:42 -05:00
asonix
f95e61e402 Expose repo migrations 2023-08-16 16:32:19 -05:00
asonix
ba27a1a223 Build out repo-repo migration 2023-08-16 16:09:40 -05:00
asonix
1559d57f0a Don't overwrite existing variants 2023-08-16 15:12:16 -05:00
asonix
22cfbe979d Note how many failures have occurred for a given hash 2023-08-16 13:29:22 -05:00
asonix
ee88823d41 clippy 2023-08-16 12:43:12 -05:00
asonix
521cdd5b9d Enforce format on new details 2023-08-16 12:36:18 -05:00
asonix
f3c6239979 Allow skipping details generation if already generated 2023-08-16 11:57:09 -05:00
asonix
4d909ba32a Make migration concurrent 2023-08-16 11:47:36 -05:00
asonix
ac996ff832 Remove unused error 2023-08-15 21:25:04 -05:00
asonix
1de257bb07 Ensure access values are unique 2023-08-15 21:18:25 -05:00
asonix
bd3b668d81 Don't error on duplicate - happens on retried migration 2023-08-15 20:28:42 -05:00
asonix
9fe7410dd6 Add logging to migration 2023-08-15 19:33:19 -05:00
asonix
1ee77b19d8 Make Repo object safe. Create migration 2023-08-15 19:19:03 -05:00
asonix
09236d731d Fix poll after completion, misused compare_and_swap 2023-08-14 21:17:57 -05:00
asonix
26ca3a7195 Remove associated Bytes type 2023-08-14 20:00:00 -05:00
asonix
da876fd553 Make it compile 2023-08-14 14:25:19 -05:00
asonix
3129f7844e BROKEN: start work on hash discriminant 2023-08-13 22:06:42 -05:00
asonix
e4e93cddf9 Return JobId from push 2023-08-13 19:52:27 -05:00
asonix
5d4486fdf4 Finish queue implementation update for sled 2023-08-13 19:47:20 -05:00
asonix
35d0c065e9 Start work on 0.5 db format - initial queuerepo changes 2023-08-13 14:12:38 -05:00
asonix
0c4d632dda Consider timeouts a client error 2023-08-05 16:18:06 -05:00
asonix
fee4ed1e3e Add process timeout 2023-08-05 12:41:06 -05:00
asonix
5f12be0c6a Clippy 2023-07-26 22:53:41 -05:00
asonix
2a59a6492a Bring back hash_alias_key 2023-07-25 20:30:22 -05:00
asonix
bd3975f455 Simplify some repo functions, remove 0.3 migration 2023-07-25 20:08:18 -05:00
asonix
f221887599 Clippy 2023-07-23 16:04:17 -05:00
asonix
aac1bb7bc4 Finish media proxy implementation 2023-07-23 15:45:52 -05:00
asonix
4cca7d0f86 Prepare for proxying media 2023-07-23 12:10:36 -05:00
asonix
75bf425c6e Update alias query, process query to accept proxy url 2023-07-23 11:45:10 -05:00
asonix
e59483c12c Add ProxyRepo 2023-07-23 10:23:46 -05:00
asonix
d0c619bdf7 Clippy 2023-07-22 21:14:24 -05:00
asonix
77cdeab57e Improve metrics, add job metrics 2023-07-22 21:11:28 -05:00
asonix
7dc8ede8eb Gate cleanup behind read-only config 2023-07-22 20:26:28 -05:00
asonix
13d6e10886 Reduce log level on variants access 2023-07-22 20:20:58 -05:00
asonix
1f3298363f Attempt to mark unmarked variants as accessed on launch 2023-07-22 19:55:50 -05:00
asonix
fe1f7c869f Enable cleanup of variants 
Still TODO: doing a first pass to mark variants as accessed on launch
 2023-07-22 19:41:50 -05:00
asonix
47e13ec04e Mark variants as accessed on fetch 2023-07-22 18:50:04 -05:00
asonix
b9e6d67d15 Add retention configuration 
This doesn't do anything yet, but is part of the Media Proxy and Variant Cleanup tickets
 2023-07-22 18:37:25 -05:00
asonix
13387dec43 Fix specs 2023-07-22 17:58:03 -05:00
asonix
b786406ad0 Change IdentifierAccess to VariantAccess 2023-07-22 17:57:52 -05:00
asonix
ce0df080f4 Add prometheus metrics 2023-07-22 16:47:59 -05:00
asonix
d661732fb4 Tidy imports 2023-07-22 12:47:32 -05:00
asonix
a390b21df1 Make running pict-rs fluent 2023-07-22 12:39:39 -05:00
asonix
3ca994ee31 Replace global config with passed-around config 
This will enable spinning up pict-rs multiple times in the same process with different configurations
 2023-07-22 12:31:01 -05:00
asonix
e1262a5fda Extract preprocess steps as urlencoded in config deserialization 2023-07-22 11:38:09 -05:00
asonix
ac48003f45 Extract ProcessMap out of static 2023-07-22 11:16:19 -05:00
asonix
8d35c2449d Simplify the cancel-safe processor 2023-07-22 10:46:16 -05:00
asonix
c1127b4117 Set content-length in PutObject, CompleteMultipart 2023-07-21 19:45:22 -05:00
asonix
2f0a3618d8 Replace awc with reqwest 2023-07-21 16:58:31 -05:00
asonix
fc14ac2038 Clippy 2023-07-18 21:59:27 -05:00
asonix
3d93300ba9 Add access repos for alias & identifier 2023-07-18 21:56:13 -05:00
asonix
127dd6cdad Make max file count configurable 2023-07-18 20:32:17 -05:00
asonix
eeac900d7e Add quality value to process method, move -coalesce earlier in pipelines 2023-07-18 16:18:01 -05:00
asonix
558605381d Fix dangling unprocessed uploads 
Adds error boundary around backgrounded ingest
 2023-07-17 22:32:08 -05:00
asonix
496bcc5dc1 Only include -b:v 0 for VP9 
also simplify building arguments
 2023-07-17 18:13:50 -05:00
asonix
3aecef3a9e Remove println 2023-07-17 17:47:48 -05:00
asonix
b1bbc6b159 Attempt adding quality settings to pict-rs 2023-07-17 17:45:26 -05:00
asonix
c4c920191f Add read-only mode 2023-07-17 14:24:49 -05:00
asonix
dd1d509bb1 Enable configuring download client timeout 2023-07-17 13:44:31 -05:00
asonix
5e8ab7856d Extract Status errors into command-specific errors 2023-07-17 13:30:08 -05:00
asonix
2961aae6e3 Port migration changes from 0.4.1 2023-07-16 22:07:42 -05:00
asonix
a751d92436 Port error changes to 0.5 2023-07-16 21:51:14 -05:00
asonix
9871ad2ee0 Clippy 2023-07-14 23:16:57 -05:00
asonix
4d78ce7689 Fix store delegation for public_url 2023-07-14 15:23:07 -05:00
asonix
786f583d98 Add public_url to Store for optional redirects to public object store urls 
Add details generation to file migration to set the content-type for uploads
 2023-07-14 14:53:37 -05:00
asonix
4e58388b41 Clean alias before checking hash option 2023-07-13 19:58:31 -05:00
asonix
afeac8294a Fix Alias cleanup ordering 
Update dev.toml
 2023-07-13 19:57:09 -05:00
asonix
ec6c9aac1e Better discovery validation, proper invoking of imagemagick 2023-07-13 19:21:57 -05:00
asonix
9b1dd1f8c3 Copy process from 0.4.x branch 2023-07-13 19:21:28 -05:00
asonix
2c22f2ee3a It compiles and runs, but doesn't work 2023-07-13 17:42:21 -05:00
asonix
27451971a6 Fix tests, add animated_webp test 2023-07-13 14:34:40 -05:00
asonix
33b83f97f2 It compiles 
Need to reinstate media limits
 2023-07-13 13:48:59 -05:00
asonix
ad1837f9dd VERY BROKEN: start replacing parts of pict-rs 2023-07-12 22:12:21 -05:00
asonix
58d9765594 WIP: Rework format detection and conversions 
- Add a bunch of types & methods that aren't used yet. The idea is
  creating the output from the known input parameters is pure, and works
  for any uploaded media
- Introduce a new kind of media "animation" which is neither an image
  nor a video. It is likely that animations can be processed properly at
  the /image/process.{ext} endpoint with a little massaging

TODO:
- Integrate this into exiftool, ffmpeg, magick commands
- Detect apng with ffmpeg (imagemagick treats it as a still png unless it's given a
    hint)
- Infallible conversion from Details to OutputFile - this might be
  impossible given that we don't currently store more than a mime type
  and a frame count to give any sort of indication of real type
- Try not to break API
 2023-07-11 23:11:25 -05:00
asonix
691bca286c Make object storage timeouts configurable 2023-07-11 13:01:58 -05:00
asonix
f5c39f9be5 Typo, error formatting 2023-07-10 17:15:43 -05:00
asonix
4be2eb67d1 Increase object storage signature expiry to 15 seconds 2023-07-10 16:28:51 -05:00
asonix
9004ecaadf Better classify process related errors 2023-07-10 15:29:41 -05:00
asonix
e10cb1401d Bump version 2023-07-09 15:16:04 -05:00
asonix
7dadbcf3ed Replace UnsupportedFormat error with more helpful errors 2023-07-09 15:07:49 -05:00
asonix
1694f49436 Use json for ffmpeg, imagemagick details parsing 2023-07-09 14:50:58 -05:00
asonix
686be1d93b Turn export_path into flag for sled 2023-07-08 19:56:26 -05:00
asonix
7768d4e58e Add database exports 
This should help admins running pict-rs on traditional filesystems to
keep valid backups of pict-rs' sled repo
 2023-07-08 17:36:01 -05:00
asonix
4680f9c3aa Instrument repo and store build methods 2023-07-08 10:42:03 -05:00
asonix
d51c348a3f Run health checks on old and new stores before migrating 2023-07-07 14:05:13 -05:00
asonix
3ccb8ecd8c Validate alias represents real file before setting not found 2023-07-07 13:33:27 -05:00
asonix
9e7376d411 Remove is_missing error, make Identifier and DeleteToken return Option on fetch 2023-07-07 13:17:26 -05:00
asonix
7274538b70 Try adding store healthcheck 2023-07-07 12:05:42 -05:00
asonix
c150903980 Expand store differentiation 2023-07-07 11:19:39 -05:00
asonix
75be876d3c Differentiate missing files in old vs new store for migration 2023-07-07 11:03:10 -05:00
asonix
508dd4340b Fix panic when migrating < 100 files, improve resumed migration messaging 2023-07-06 16:06:37 -05:00
asonix
3487cb0e30 Add ability to set 404 image 
Fix imagemagick blur arguments
 2023-07-05 19:07:03 -05:00
asonix
e7e4876908 Explicitly handle cases of invalid aliases 2023-07-05 16:46:44 -05:00
asonix
23e67b9697 Enhance migration handling of missing identifiers 
Queue a cleanup task for hashes that don't have original file identifiers
Be more specific about what field is missing when a field is missing
 2023-07-05 09:52:19 -05:00
asonix
2274dfecb4 Fix io-uring - read_to_end was polling Stream mapped through StreamReader after completion 2023-07-05 00:35:07 -05:00
asonix
c8e84b8914 Greatly improve error information during migration 2023-07-02 21:07:15 -05:00
asonix
e6f17cbac6 Log on successful migration 2023-06-29 11:45:06 -05:00
asonix
01d3610f46 Add basic progress indication for store migration 2023-06-29 11:39:47 -05:00
asonix
0d1df5ebc2 Avoid failed identifier cleanup on failed duplicate upload 2023-06-23 11:43:15 -05:00
asonix
7ba2fc9b26 Fix duplicate uploads potentially purging existing uploads on failure 2023-06-23 11:39:43 -05:00
asonix
58f0c328d9 Fix typo in error message 2023-06-23 11:20:55 -05:00
asonix
a1d48cbb42 Share client between object store and request extractor 
Add config to limit connection pool size
 2023-06-23 11:20:20 -05:00
asonix
071d788349 Include original error in object storage sendrequest error 2023-06-21 17:07:14 -05:00
asonix
a3ff072350 Remove unused cache duration parameter 2023-06-21 17:05:53 -05:00
asonix
6d2aef8cc0 Add jpegxl and avif support 2023-06-21 17:05:35 -05:00
asonix
554d852e68 Create unique errors for Repo and Store, separate from UploadError 
Add .is_not_found() for Store errors, wire up Object and File storage to properly set NotFound
Allow skipping files that are not found during store migration
 2023-06-20 15:59:08 -05:00
asonix
1214b51ad7 Make use of FilesystemDefaults when constructing migrate commands 2023-06-19 15:04:36 -05:00
asonix
a3a986638d Improve gif handling. 
- Allow a fast-path exif cleaning if we'd re-encode a gif to a gif
- Use single-frame palettes to better map transparency from frame to frame

Unrelated:
- decrease ffmpeg logging when creating thumbnails
 2023-06-19 14:25:31 -05:00
asonix
22e526fd2a Allow type underscore (probably in instrument) 2023-06-01 17:33:43 -05:00
asonix
cc314c37a6 Use libopus for ffmpeg 2023-03-09 21:29:56 -06:00