mirrors/pict-rs
1
0
Fork 0
mirror of https://git.asonix.dog/asonix/pict-rs.git synced 2024-12-04 15:26:27 +00:00
Code Issues Projects Releases Packages Wiki Activity
1350 commits 27 branches 313 tags 6 MiB
Commit graph

624 commits

Author SHA1 Message Date
asonix
2961aae6e3 Port migration changes from 0.4.1 2023-07-16 22:07:42 -05:00
asonix
a751d92436 Port error changes to 0.5 2023-07-16 21:51:14 -05:00
asonix
9871ad2ee0 Clippy 2023-07-14 23:16:57 -05:00
asonix
4d78ce7689 Fix store delegation for public_url 2023-07-14 15:23:07 -05:00
asonix
786f583d98 Add public_url to Store for optional redirects to public object store urls 
Add details generation to file migration to set the content-type for uploads
 2023-07-14 14:53:37 -05:00
asonix
4e58388b41 Clean alias before checking hash option 2023-07-13 19:58:31 -05:00
asonix
afeac8294a Fix Alias cleanup ordering 
Update dev.toml
 2023-07-13 19:57:09 -05:00
asonix
ec6c9aac1e Better discovery validation, proper invoking of imagemagick 2023-07-13 19:21:57 -05:00
asonix
9b1dd1f8c3 Copy process from 0.4.x branch 2023-07-13 19:21:28 -05:00
asonix
2c22f2ee3a It compiles and runs, but doesn't work 2023-07-13 17:42:21 -05:00
asonix
27451971a6 Fix tests, add animated_webp test 2023-07-13 14:34:40 -05:00
asonix
33b83f97f2 It compiles 
Need to reinstate media limits
 2023-07-13 13:48:59 -05:00
asonix
ad1837f9dd VERY BROKEN: start replacing parts of pict-rs 2023-07-12 22:12:21 -05:00
asonix
58d9765594 WIP: Rework format detection and conversions 
- Add a bunch of types & methods that aren't used yet. The idea is
  creating the output from the known input parameters is pure, and works
  for any uploaded media
- Introduce a new kind of media "animation" which is neither an image
  nor a video. It is likely that animations can be processed properly at
  the /image/process.{ext} endpoint with a little massaging

TODO:
- Integrate this into exiftool, ffmpeg, magick commands
- Detect apng with ffmpeg (imagemagick treats it as a still png unless it's given a
    hint)
- Infallible conversion from Details to OutputFile - this might be
  impossible given that we don't currently store more than a mime type
  and a frame count to give any sort of indication of real type
- Try not to break API
 2023-07-11 23:11:25 -05:00
asonix
691bca286c Make object storage timeouts configurable 2023-07-11 13:01:58 -05:00
asonix
f5c39f9be5 Typo, error formatting 2023-07-10 17:15:43 -05:00
asonix
4be2eb67d1 Increase object storage signature expiry to 15 seconds 2023-07-10 16:28:51 -05:00
asonix
9004ecaadf Better classify process related errors 2023-07-10 15:29:41 -05:00
asonix
e10cb1401d Bump version 2023-07-09 15:16:04 -05:00
asonix
7dadbcf3ed Replace UnsupportedFormat error with more helpful errors 2023-07-09 15:07:49 -05:00
asonix
1694f49436 Use json for ffmpeg, imagemagick details parsing 2023-07-09 14:50:58 -05:00
asonix
686be1d93b Turn export_path into flag for sled 2023-07-08 19:56:26 -05:00
asonix
7768d4e58e Add database exports 
This should help admins running pict-rs on traditional filesystems to
keep valid backups of pict-rs' sled repo
 2023-07-08 17:36:01 -05:00
asonix
4680f9c3aa Instrument repo and store build methods 2023-07-08 10:42:03 -05:00
asonix
d51c348a3f Run health checks on old and new stores before migrating 2023-07-07 14:05:13 -05:00
asonix
3ccb8ecd8c Validate alias represents real file before setting not found 2023-07-07 13:33:27 -05:00
asonix
9e7376d411 Remove is_missing error, make Identifier and DeleteToken return Option on fetch 2023-07-07 13:17:26 -05:00
asonix
7274538b70 Try adding store healthcheck 2023-07-07 12:05:42 -05:00
asonix
c150903980 Expand store differentiation 2023-07-07 11:19:39 -05:00
asonix
75be876d3c Differentiate missing files in old vs new store for migration 2023-07-07 11:03:10 -05:00
asonix
508dd4340b Fix panic when migrating < 100 files, improve resumed migration messaging 2023-07-06 16:06:37 -05:00
asonix
3487cb0e30 Add ability to set 404 image 
Fix imagemagick blur arguments
 2023-07-05 19:07:03 -05:00
asonix
e7e4876908 Explicitly handle cases of invalid aliases 2023-07-05 16:46:44 -05:00
asonix
23e67b9697 Enhance migration handling of missing identifiers 
Queue a cleanup task for hashes that don't have original file identifiers
Be more specific about what field is missing when a field is missing
 2023-07-05 09:52:19 -05:00
asonix
2274dfecb4 Fix io-uring - read_to_end was polling Stream mapped through StreamReader after completion 2023-07-05 00:35:07 -05:00
asonix
c8e84b8914 Greatly improve error information during migration 2023-07-02 21:07:15 -05:00
asonix
e6f17cbac6 Log on successful migration 2023-06-29 11:45:06 -05:00
asonix
01d3610f46 Add basic progress indication for store migration 2023-06-29 11:39:47 -05:00
asonix
0d1df5ebc2 Avoid failed identifier cleanup on failed duplicate upload 2023-06-23 11:43:15 -05:00
asonix
7ba2fc9b26 Fix duplicate uploads potentially purging existing uploads on failure 2023-06-23 11:39:43 -05:00
asonix
58f0c328d9 Fix typo in error message 2023-06-23 11:20:55 -05:00
asonix
a1d48cbb42 Share client between object store and request extractor 
Add config to limit connection pool size
 2023-06-23 11:20:20 -05:00
asonix
071d788349 Include original error in object storage sendrequest error 2023-06-21 17:07:14 -05:00
asonix
a3ff072350 Remove unused cache duration parameter 2023-06-21 17:05:53 -05:00
asonix
6d2aef8cc0 Add jpegxl and avif support 2023-06-21 17:05:35 -05:00
asonix
554d852e68 Create unique errors for Repo and Store, separate from UploadError 
Add .is_not_found() for Store errors, wire up Object and File storage to properly set NotFound
Allow skipping files that are not found during store migration
 2023-06-20 15:59:08 -05:00
asonix
1214b51ad7 Make use of FilesystemDefaults when constructing migrate commands 2023-06-19 15:04:36 -05:00
asonix
a3a986638d Improve gif handling. 
- Allow a fast-path exif cleaning if we'd re-encode a gif to a gif
- Use single-frame palettes to better map transparency from frame to frame

Unrelated:
- decrease ffmpeg logging when creating thumbnails
 2023-06-19 14:25:31 -05:00
asonix
22e526fd2a Allow type underscore (probably in instrument) 2023-06-01 17:33:43 -05:00
asonix
cc314c37a6 Use libopus for ffmpeg 2023-03-09 21:29:56 -06:00
asonix
dc27200a35 Add transparency to gifs, rustfmt 2023-02-25 14:31:57 -06:00
asonix
97ba4e806b Clippy 2023-02-25 13:01:55 -06:00
asonix
9a2f846cd1 Detect input transparency for gif/webm 2023-02-25 12:59:47 -06:00
asonix
281ac43dff Remove cache functionality 2023-02-25 11:34:48 -06:00
asonix
123488f617 clippy 2023-02-12 23:18:27 -06:00
asonix
788b87eceb Remove obsolute pre-0.3 migrate check 2023-02-12 23:13:47 -06:00
asonix
138499d47f Use transpose in a couple places 2023-02-05 20:50:59 -06:00
asonix
cc75d2d882 Clippy 2023-02-04 17:54:56 -06:00
asonix
4cc810d372 Add control over gif frame count 2023-02-04 17:52:23 -06:00
asonix
88ca9793e8 s/vrames/frames:v 2023-02-04 17:47:03 -06:00
asonix
40f57be0c7 Allow uploading small gifs 2023-02-04 17:32:36 -06:00
asonix
0aa3f574a5 Clippy 2023-01-29 11:57:59 -06:00
asonix
f6d6d54b88 Update deps 2023-01-29 11:47:28 -06:00
asonix
f0f40db8c3 Add healthcheck for db 2023-01-29 11:36:09 -06:00
asonix
5e5dd27a05 Retry migrate_file, remove variant before relating new one 2023-01-04 18:58:05 -06:00
asonix
28f7a139a0 Retry in-process, and rate-limit self 2023-01-04 17:26:05 -06:00
asonix
c55b1115d1 Increase client timeout, attempt to keep track of migration progress for resuming 2023-01-04 15:58:32 -06:00
asonix
dabcc24f98 Parse environment variables - main app 2023-01-04 15:05:34 -06:00
asonix
a489411c6c Parse environment variables 2023-01-04 14:42:04 -06:00
asonix
3c844d86c5 Allow configuring pict-rs through serializable types 2022-12-26 16:35:25 -06:00
asonix
2c1a37fb62 Extract pict-rs as library 2022-12-23 12:56:15 -06:00
asonix
74cd3392fa Borrow parent spans when creating children 2022-12-07 22:43:12 -06:00
asonix
3325b222a2 clippy 2022-11-26 11:18:34 -06:00
asonix
7037ee5791 Support auto-orienting photos 2022-10-15 11:13:24 -05:00
asonix
669b3fb86f Fix webp metadata stripping, more tracing cleanup 2022-10-01 22:47:52 -05:00
asonix
718f09c43a Clean tracing, simplify validation, rename InputFormat -> VideoFormat 2022-10-01 21:17:18 -05:00
asonix
e7cf21f862 Allow concurrency in small object uploads 2022-10-01 13:06:19 -05:00
asonix
fe2a499110 Fix sending upload parts (set size), optimize small file uploads 2022-10-01 13:00:07 -05:00
asonix
0307dc5a3b Introduce BytesStream to simplify aggregating, forwarding bytes 2022-10-01 12:06:33 -05:00
asonix
c767804430 Default to vp9 for uploaded video 2022-10-01 10:17:32 -05:00
asonix
85c5ce955d Clippy 2022-09-30 20:02:46 -05:00
asonix
d86d9a3228 Put proper file extension on videos 2022-09-30 20:00:14 -05:00
asonix
8eb2293808 Enable selecting video and audio codecs for uploaded media 2022-09-30 19:38:11 -05:00
asonix
f4542efcc1 Share ffprobe details code 2022-09-30 17:43:40 -05:00
asonix
3ceaaa391c Update clap 2022-09-28 18:23:41 -05:00
asonix
656b0225af Clippy 2022-09-27 23:23:06 -05:00
asonix
29cab025c6 Improve 0.3 migration code, repo traces 2022-09-27 23:19:52 -05:00
asonix
85f20bbe0e Add new resize options, update README with current APIs 2022-09-26 18:14:25 -05:00
asonix
9db057fdc5 Add admin endpoint to fetch identifier from an alias 2022-09-25 20:59:57 -05:00
asonix
17dab63662 Clippy 2022-09-25 20:44:24 -05:00
asonix
80c83eb491 Add ffprobe for details inspection - vastly improve video detection speed 2022-09-25 20:39:09 -05:00
asonix
5449bb82f1 Better identify video uploads with ffprobe 2022-09-25 19:34:51 -05:00
asonix
21e3e63ac3 Instrument upload store methods 2022-09-25 18:46:50 -05:00
asonix
51d8621015 Defensively format errors in traces to avoid deadlocks 2022-09-25 18:46:26 -05:00
asonix
890478e794 Support audio in uploaded videos, allow webm uploads 2022-09-25 18:16:37 -05:00
asonix
c57a48db8a Start introducing more video controls 2022-09-25 17:36:07 -05:00
asonix
c9a74a73ca Reduce redundancy in process module 2022-09-25 17:35:52 -05:00
asonix
50d118a2a7 Implement image pre-processing on upload 2022-09-25 15:17:33 -05:00
asonix
04bc586a26 Fix hasher test 2022-09-25 09:09:05 -05:00
asonix
dc95417846 Improve concurrency for upload streams part 2 2022-09-25 08:27:11 -05:00
asonix
bf3c47e457 Improve concurrency for upload streams 2022-09-24 22:07:06 -05:00
asonix
a6adde874e Implement s3 multipart uploads 2022-09-24 20:33:59 -05:00
asonix
25209e29c0 Make it compile 2022-09-24 17:18:53 -05:00
asonix
ff1771e016 More migration work for rusty-s3 2022-09-24 14:18:49 -05:00
asonix
ab7fd9aaf7 Start work on using rusty-s3 instead of rust-s3 2022-09-24 13:39:27 -05:00
asonix
c5b107eb4c clippy 2022-09-11 10:04:37 -05:00
asonix
7f49af58fe Update actix-form-data, impl Store & Repo for Data<T> 2022-09-10 19:29:22 -05:00
asonix
e0bc95f785 Remove unneeded reborrow 2022-08-15 15:00:12 -05:00
Aode (lion)
547934071a Don't migrate from the old database if it doesn't exist 2022-08-14 11:49:20 -05:00
Aode (lion)
e0543b2c1c Add HEAD endpoints for process, serve 2022-07-17 11:15:43 -05:00
asonix
cb9a594f24 Fix content type on already processed images 2022-06-21 11:27:22 -05:00
asonix
dc73bdbad4 Make already-claimed a 400 2022-06-05 23:24:58 -05:00
Aode (lion)
2db76ad8c5 Update dependencies 2022-04-29 15:54:54 -05:00
Aode (lion)
8ee26ccf4f change some import paths 2022-04-27 20:13:49 -05:00
Aode (Lion)
4ac20546ce Improve migration from 0.3 formats 2022-04-24 17:10:15 -05:00
Aode (Lion)
b3cbda1337 Add internal endpoint for cleaning variants 2022-04-11 16:56:39 -05:00
Aode (Lion)
e50ccfb48c Update deps 2022-04-11 15:17:54 -05:00
Aode (lion)
05533f7e3a Use rfc3339 dates for details responses 2022-04-08 13:36:06 -05:00
Aode (lion)
7436f15267 Dry download client logic 2022-04-08 13:16:22 -05:00
Aode (lion)
5f19b63166 Extract details processing 2022-04-08 13:03:00 -05:00
Aode (lion)
31c5a36c77 Better instrument drops, jobs. Properly disarm backgrounded downloads 2022-04-08 12:51:33 -05:00
Aode (lion)
55d5e43cd2 Consolidate endpoints for downloading media, cached or backgrounded 2022-04-08 12:05:14 -05:00
Aode (lion)
07c61cbe87 Add download backgrounded 2022-04-07 19:07:30 -05:00
Aode (lion)
8781bc8f28 Add follows-from relation for background tasks 2022-04-07 13:28:28 -05:00
Aode (lion)
e493e90dd4 Defensive tracing 2022-04-07 12:56:40 -05:00
Aode (lion)
c80d207a87 Improve tracing 2022-04-06 21:40:49 -05:00
Aode (Lion)
05e2cf5e08 Cache on Alias rather than Hash 
Make sled repo blocking tasks share parent span
 2022-04-06 12:14:12 -05:00
Aode (Lion)
865f4ee09c Caching works: 
- Insert new bucket for current access time before removing
    previous access time record.
- Add instrument to more methods for more precise tracing
 2022-04-05 21:47:35 -05:00
Aode (Lion)
6cdae7b318 Add 'cache' endpoint for ingesting ephemeral media 
By default, cached media should only stick around for 7 days, however
The timeout is reset every time media is accessed, so only obscure
cached media will be flushed from the cache. This '7 days' number is
configurable through the commandline run options as --media-cache-duration
and in the pict-rs.toml file as [media] cache_duration
 2022-04-05 20:29:30 -05:00
Aode (lion)
b0670c2f66 Move migrate to old 2022-04-03 15:07:31 -05:00
Aode (lion)
409b7d4c54 Remove unused conversions 2022-04-03 12:51:21 -05:00
Aode (lion)
29f0774331 Add ability to claim uploads 2022-04-02 21:15:39 -05:00
Aode (lion)
8734dfbdc7 Add backgrounding routes 
- Accept backgrounded uploads
- Allow backgrounded processing

Still TODO:
- Endpoint for waiting on/claiming an upload
 2022-04-02 20:56:29 -05:00
Aode (Lion)
c4d014597e Better job recovery 2022-04-02 18:53:03 -05:00
Aode (Lion)
77a400c7ca Implement UploadRepo 2022-04-02 17:41:00 -05:00
Aode (Lion)
132e395e5c Multiple items: 
- Reduce duplicate work in generate job
- Use hash instead of identifier for unique processing
- Move motion ident generation behind concurrent processor lock
 2022-04-02 17:40:08 -05:00
Aode (Lion)
09f53b9ce6 Move away from UploadManager to direct repo & store actions 2022-04-02 16:44:03 -05:00
Aode (lion)
6ed592c432 In Progress: process jobs 
- Is this a good idea? it definitely will make interacting with pict-rs harder. Maybe it's best not to do this
 2022-04-01 16:51:12 -05:00
Aode (lion)
c0d8e0e8e3 Support multiple queues for job processor 2022-04-01 11:51:46 -05:00
Aode (lion)
09281d9ae8 Move stream timeout to all response bodies 2022-03-29 16:48:26 -05:00
Aode (lion)
63d66050c8 Enable buffering iterator items for iterstream 2022-03-29 16:18:00 -05:00
Aode (lion)
5adb3fde89 Move some Stream implementations into 'stream' module 2022-03-29 15:59:17 -05:00
Aode (lion)
0e490ff54a Remove custom BytesFreezer stream 2022-03-29 14:28:08 -05:00
Aode (lion)
25f4480809 Remove custom stream error mapping types 2022-03-29 14:17:20 -05:00
Aode (lion)
8226a3571d Enable multi-threaded job processing 2022-03-29 13:18:47 -05:00
Aode (lion)
602d1ea935 Begin implementing queue, put cleanups in it 2022-03-29 12:51:16 -05:00
Aode (lion)
e25a4781a8 Remove unneeded variable 2022-03-29 11:09:23 -05:00
Aode (lion)
99d042f36e Allow disabling gif/mp4 2022-03-29 11:04:56 -05:00
Aode (Lion)
eb5e39c634 Use color-eyre 2022-03-28 20:47:46 -05:00
Aode (lion)
1291bf8beb Divorce reqwest from main application's spans 2022-03-28 18:42:22 -05:00
Aode (lion)
1662f153ce Numerous changes: 
- Improve error printing (display chain in addition to spantrace)
- Fix migration (read main identifier from identifier tree, not filename tree)
- Ensure uniqueness for processed images in ConcurrentProcessor (use source identifier in addition to thumbnail path, include extension in thumbnail path)
- Update default log levels (make pict-rs quieter)
- Add timeout for serving images from object storage (5 seconds)
 2022-03-28 15:34:36 -05:00
Aode (Lion)
3792a8923a Finish redoing config probably 2022-03-27 23:27:07 -05:00
Aode (lion)
ca28f68ef5 Start redoing config again 2022-03-27 19:10:06 -05:00
Aode (lion)
37e6b21b55 Fix Range, consolidate errors, test object storage 2022-03-26 20:45:12 -05:00
Aode (lion)
15b52ba6ec Get it compiling again 2022-03-26 16:49:23 -05:00
Aode (lion)
323016f994 Build out migration path 2022-03-25 18:47:50 -05:00
Aode (lion)
750ce4782e Rework configuration 2022-03-24 22:06:29 -05:00
Aode (lion)
d6567fbbbd Implement sled repo 2022-03-24 17:09:15 -05:00
Aode (lion)
a0c99d05eb Create initial trait representation for database 2022-03-24 13:16:41 -05:00
Aode (Lion)
2ad536db17 Promote console from compile flag to runtime flag 2022-03-21 22:41:51 -05:00
Aode (Lion)
f13f870a92 Only use console config when console feature is enabled 2022-03-21 22:05:32 -05:00
Aode (Lion)
62615be371 Allow configuring console buffer capacity 2022-03-21 21:56:19 -05:00
Aode (Lion)
d30415bb49 Don't debug PathStore 2022-03-21 21:43:38 -05:00
Aode (Lion)
9a8dc0e44f Fix sled cache capacity help text 2022-03-21 21:18:51 -05:00
Aode (Lion)
da159c69cb Try setting a lower default cache capacity 2022-03-21 21:15:43 -05:00
Aode (lion)
21b6e1dc08 Use actix-web's Range header 2022-03-01 11:23:15 -06:00
Aode (Lion)
73137ede21 Release candidate 1 2022-02-26 12:22:30 -06:00
Aode (lion)
0e948a2bf5 Add flag for console 2022-02-01 10:10:52 -06:00
Aode (Lion)
445c99dbf1 Re-use existing BytesMut when streaming file 2022-01-13 18:11:46 -06:00
Aode (lion)
f79d38df8d Update rust-runner image, improve tokio-console support 2021-12-19 12:59:51 -06:00
Aode (lion)
c48808c1e1 Update deps 2021-12-18 20:24:22 -06:00
Aode (lion)
44d171252b console 2021-12-16 20:46:46 -06:00
Aode (lion)
c445615c47 Update to latest betas 2021-12-12 14:48:37 -06:00
Aode (lion)
e209a282d9 No more follows-from for process 2021-12-06 21:11:43 -06:00
Aode (lion)
11739cd42c Remove fancy span following code 2021-12-06 20:35:22 -06:00
Aode (lion)
a47f2a3428 Log on new span, no pretty 2021-12-06 18:48:07 -06:00
Aode (lion)
920f1f154c Improve follows-from and parent span relationships 2021-12-06 12:43:03 -06:00
Aode (lion)
42aed5165c Update tracing-awc, add drone 2021-12-03 14:21:31 -06:00
Aode (lion)
7a3d1bef5a Only log closing spans, not opening spans 2021-11-24 21:54:27 -06:00
Aode (lion)
5f5c1d3530 Merge branch 'main' of git.asonix.dog:asonix/pict-rs into main 2021-11-23 16:31:19 -06:00
Aode (lion)
5b44de2070 clippy 2021-11-23 16:31:15 -06:00
Aode (Lion)
79afd0e8ea Update deps 2021-11-23 12:29:42 -06:00
Aode (lion)
0d153ca239 Make stores take filename into account when saving files 2021-11-16 09:45:16 -06:00
Aode (lion)
c5d6610276 Properly handle out-of-bounds range requests 2021-11-03 12:04:59 -05:00
Aode (lion)
4c8da2b414 Keep a reqwest client around for sharing an object storage connection pool 2021-11-02 17:21:00 -05:00
Aode (lion)
ad7c323985 Update readme, cleanup migration key on completion 2021-11-01 10:44:49 -05:00
Aode (lion)
93c268fc58 Flush at every migrated file 2021-11-01 10:38:11 -05:00
Aode (lion)
ccb9f49b8b Implement storage migration 2021-10-31 21:11:35 -05:00
Aode (lion)
cf50da30ec Use rust-s3 fork with asyncwrite support 2021-10-31 12:35:11 -05:00
Aode (lion)
76e3d37009 Clean up ObjectStore's Debug impl 2021-10-28 21:07:31 -05:00
Aode (lion)
366db1e1fe Make object storage with minio work 2021-10-28 20:59:11 -05:00
Aode (lion)
657152e1a8 Don't println 2021-10-28 00:18:18 -05:00
Aode (lion)
f596edc363 Improve configuration documentation 2021-10-28 00:17:37 -05:00
Aode (lion)
f9f4fc63d6 Optionally support s3-compatible storage (untested) 2021-10-27 23:06:03 -05:00
Aode (lion)
26a2401027 Use tmp files for mp4s 2021-10-23 14:14:12 -05:00
Aode (lion)
6f04595c3b Remove unneeded mime conversions 2021-10-23 12:35:07 -05:00
Aode (lion)
36b83f114a Merge branch 'main' into asonix/generic-pictrs-storage 2021-10-22 23:49:47 -05:00
Aode (lion)
48557bc2ea Make pict-rs generic over file storage 2021-10-22 23:48:56 -05:00
Aode (Lion)
2295837ea4 2021 2021-10-21 16:36:31 -05:00
Aode (Lion)
f67aeb92fa Keep better track of gif/mp4 thumbnail 2021-10-20 21:36:18 -05:00
Aode (Lion)
1767942e31 Remove recursion 2021-10-20 20:36:10 -05:00
Aode (Lion)
d69d261999 Make it actually compilable 2021-10-20 20:30:21 -05:00
Aode (Lion)
4aefc0403a Remove more boxes 2021-10-20 20:13:39 -05:00
Aode (Lion)
877390878b Remove more boxes 2021-10-20 19:28:40 -05:00
Aode (Lion)
e7c3e2c96c Rename stream module 2021-10-20 19:00:41 -05:00
Aode (Lion)
e7b4e4d1cc Remove a few boxes 2021-10-20 18:58:32 -05:00
Aode (Lion)
922200673e Update to latest betas 2021-10-20 18:16:53 -05:00
Aode (Lion)
9559a8b202 Update cli flags 2021-10-18 23:51:04 -05:00
Aode (Lion)
3fe66129be Update opentelemetry url info 2021-10-18 23:44:56 -05:00
Aode (Lion)
34038a0f16 Fix restructure, upload 2021-10-18 23:37:11 -05:00
Aode (Lion)
195f614a4b Restructure attempt 1 2021-10-18 20:29:06 -05:00
Aode (Lion)
0f3be6566e Split upload_manager files 2021-10-18 18:44:05 -05:00
Aode (Lion)
2fb8d3e39a Reduce required Unpin bounds with pin-project-lite 2021-10-18 18:02:33 -05:00
Aode (lion)
4c5482b3a8 Add more instrumentation 2021-10-13 20:31:40 -05:00
Aode (lion)
786d8469ee It works 2021-10-13 19:50:07 -05:00
Aode (Lion)
09cb2a53b0 Rewrite to avoid direct AsyncX impls 2021-10-13 19:06:53 -05:00
Aode (lion)
2318fd9dca Enable io-uring 2021-10-12 23:16:31 -05:00
Aode (Lion)
23e6657208 Re-enable skipping validation on import, strip when converting to prescribed format 2021-09-26 11:02:19 -05:00
Aode (Lion)
15dc79d10e Instrument a couple more functions 2021-09-25 17:09:55 -05:00
Aode (lion)
f390c62cbf Wrap BytesFreezer in span 2021-09-25 16:39:10 -05:00
Aode (lion)
ba68bcbde3 Add better span information to commands, spawned tasks 2021-09-25 15:23:05 -05:00
Aode (lion)
7cb6a15acd Wire up cross-application propagation 2021-09-19 14:36:53 -05:00
Aode (lion)
79a0fa32ee Remove tracing-error workaround 2021-09-18 16:34:23 -05:00
Aode (lion)
5e9d923d03 Instrument for OpenTelemetry 2021-09-18 16:29:30 -05:00
Aode (lion)
06e402ec72 Don't enter across await points 2021-09-16 17:51:20 -05:00
Aode (Lion)
acba660c37 Update to latest actix-form-data 2021-09-13 20:50:51 -05:00
Aode (Lion)
5d3e6f50b3 Instrument better with Tracing 2021-09-13 20:22:42 -05:00
Aode (lion)
79bd562d8e Clippy lints 2021-09-12 10:42:44 -05:00
Aode (lion)
68e4f1f4c8 Make uploads cancel-safe 2021-09-11 19:53:26 -05:00
Aode (lion)
bf1a16d7d3 Abort process writer task if reader is dropped 2021-09-11 16:35:38 -05:00
Aode (lion)
0ffd66c74e Add and document request deadlines 2021-09-11 15:31:00 -05:00
Aode (Lion)
c1b2c7706a Add back range header support for thumbnail on generation 2021-09-09 16:35:07 -05:00
Aode (Lion)
45606f4579 Add application-level dimension limiting, bail on failed processes 2021-09-09 14:16:12 -05:00
Aode (lion)
a0f7148924 Cargo fmt 2021-09-06 12:09:25 -05:00
Aode (lion)
0d5ce07b20 Convert OnceCell to Lazy, fmt 2021-09-06 12:08:13 -05:00
Aode (Lion)
ac79a9d5cd Use futures-util instead of hand-implemented types 2021-09-05 20:00:31 -05:00
Aode (Lion)
80044616f9 Make CancelSafeProcessor impl more concise, format code 2021-09-05 19:05:58 -05:00
Aode (Lion)
810831ca73 Don't process images that are already being processed 2021-09-05 17:15:30 -05:00
Aode (Lion)
e88b9a024a Add semaphore around process operations 2021-09-04 16:01:48 -05:00
Aode (lion)
e21fd29c09 Drop unneeded dependencies 2021-09-04 14:20:31 -05:00
Aode (lion)
08c3169d3f Fewer streams 2021-09-04 12:42:40 -05:00
Aode (lion)
1119ed740e WIP: Try to pass AsyncRead types around instead of Streams 2021-09-03 19:53:53 -05:00
Aode (lion)
43f94a3d5d Fix imagemagick policy for gifs 2021-08-31 12:14:23 -05:00
Aode (lion)
ee215bfa5e Version v0.3.0-alpha.21-in-mem 2021-08-31 11:02:30 -05:00
Aode (lion)
a6f2082b37 Spawn new task to avoid complexity/bugs 2021-08-31 10:17:08 -05:00
Aode (Lion)
c1d4e3b87e Start work on piping bytes around from memory instead of going to disk and back 2021-08-30 21:19:47 -05:00
Aode (lion)
3578303104 Version v0.3.0-alpha.19 2021-08-29 14:56:28 -05:00
Aode (lion)
18c6481fff Print outfile when converting / processing 2021-08-29 14:16:55 -05:00
Aode (lion)
3066e4d350 Use safe_move_file to create thumbnail image 2021-08-29 12:12:22 -05:00
Aode (lion)
98ec2c5a90 Magick on tmp files, allow tmp files in policy 2021-08-29 11:51:56 -05:00
Aode (lion)
91207e1a6f Version v0.3.0-alpha.15 2021-08-29 00:05:02 -05:00
Aode (lion)
7c371c182a Version v0.3.0-alpha.14 2021-08-28 23:15:29 -05:00
Aode (lion)
db95c85bdf Version v0.3.0-alpha.13-shell-out-r4 2021-08-28 22:05:49 -05:00
Aode (lion)
d4733321ed Version v0.3.0-alpha.13-shell-out-r3 2021-08-28 20:37:53 -05:00
Aode (lion)
edd6bb4a60 Replace all bindings with calls to external binaries 2021-08-28 17:15:14 -05:00
Aode (Lion)
7fd707c8df Begin work on spawning processes rather than binding to c libs 2021-08-25 21:46:11 -05:00
asonix
a2c17a950b Update actix deps 2021-06-26 13:10:28 -05:00
asonix
120184dd0c Update dependencies 2021-06-19 14:39:41 -05:00
asonix
5e452ef27e Update actix deps 2021-04-17 14:59:42 -05:00
asonix
7aa32b9b3b Update actix deps 2021-03-09 20:51:03 -06:00
asonix
79b26a1dda Update to actix-web 4.0.0-beta.3 2021-02-10 16:57:42 -06:00
asonix
9797e2e3fc Go back to actix-fs 2021-01-18 19:54:39 -06:00
asonix
8c1f60101b Add module for Range Header operations 
Switch from actix-fs to async-fs
 2021-01-18 17:11:32 -06:00
John Doe
da237eacae fmt 2021-01-14 11:26:17 -05:00
John Doe
c663337bcd ranged support for process endpoint, bugfixes 2021-01-14 08:52:11 -05:00
John Doe
61061f0451 range header support 2021-01-13 20:08:04 -05:00
asonix
c0520d181b Randomize tmp folder name, clean tmp folder on exit 2021-01-05 11:51:13 -06:00
asonix
2e9fb2a830 Update download route to return 'details' 2021-01-05 00:28:17 -06:00
asonix
0acdfffe08 Return details in upload method, clippy nits 2020-12-25 20:39:19 -06:00
asonix
a750364e27 Expose cropping 2020-12-16 18:20:06 -06:00
asonix
c0a20588d3 Add content type, width, height to Details, add details endpoints 2020-12-10 12:49:10 -06:00
asonix
0bf7e0f688 Add initial details implementation 
Details is an object generated for each image, containing
it's created_at date to properly set response headers. It
also can be expanding in the future to include an image's
dimensions, which we can expose via an API endpoint.

In addition to this, Details allows us the ability to
re-generate processed images. For now, all images which
do not yet have an associated Details will be
re-generated upon access. Details is set for each
thumbnail on generation, so this will only happen once
for each image.
 2020-12-09 23:05:07 -06:00
asonix
4bd99431e2 Generate image bytes based on request format 2020-11-01 14:25:21 -06:00
asonix
4ce7b79b86 Safely create parent tmp dir when needed 2020-10-16 17:48:42 -05:00
asonix
b64e3615be Switch many migration logs to debug, flush changes to new db 2020-10-16 16:02:48 -05:00
asonix
577d358da1 Enhance migrations to be more resilient to panics 2020-10-16 15:28:45 -05:00
asonix
f4f1ae6939 Update migration string 2020-10-10 19:38:07 -05:00
asonix
04e5d45679 Add Last-Modified header, extend MaxAge cache 2020-10-02 15:38:52 -05:00
asonix
6de19d17b6 Clean migration, don't use db tree 2020-09-14 16:42:31 -05:00
asonix
2515de00c2 Stable actix 2020-09-14 10:52:18 -05:00
asonix
4a3bd42c14 ffmpeg features, sled 034 2020-09-10 11:12:42 -05:00
asonix
7c0a407568 Add privileged purging, alias <-> filename relation queries 2020-07-11 17:22:45 -05:00
asonix
fd809e4a0b Make 'import' a protected method 2020-07-11 16:28:49 -05:00
asonix
de3e04a411 Support mp4 2020-07-11 16:01:09 -05:00
asonix
d28d56fb9a Fix odd-pixeled gif conversion 2020-07-11 15:49:33 -05:00
asonix
510f955cc0 Migrate from 0.32.0-rc1 2020-07-11 15:40:40 -05:00
asonix
cc37f0dd8a Remove redundant default info, update readme help, add 'check' helper 2020-06-25 18:17:05 -05:00
asonix
a746793187 Enable mp4 with ffmpeg 2020-06-24 11:58:46 -05:00
asonix
eb685f99e7 Use sample for thumbnail generation 2020-06-16 21:05:06 -05:00
asonix
6dab3b30e8 Purge image 2020-06-16 18:13:22 -05:00
asonix
0fef7abc67 Validate with magickwand 2020-06-16 16:34:37 -05:00
asonix
df85faff8b Start work on webp support 2020-06-16 15:55:24 -05:00
asonix
0801c1141f dev build works with imagemagick 2020-06-15 14:20:25 -05:00
asonix
84c10edd0e Improve bounds check, don't save variant unless it's been changed 2020-06-14 23:10:30 -05:00
asonix
928fb8f277 Replace 'move' with 'copy+delete' 2020-06-14 22:35:00 -05:00
asonix
154914e61a Use rexiv2 for metadata removal 2020-06-14 21:41:45 -05:00
asonix
24f09408fb Further clean tracing logs 2020-06-14 13:56:42 -05:00
asonix
3b4c5b646a Do better logging 2020-06-14 10:07:31 -05:00
asonix
68721fb269 Replace log with tracing, only hardcode 'info' if RUST_LOG is empty 2020-06-13 21:28:06 -05:00
asonix
53345823bb Better handle upload errors 2020-06-13 14:04:33 -05:00
asonix
6de89a3318 Don't use image crate to validate gifs 2020-06-11 11:46:00 -05:00
asonix
bfdfb737ed Update deps 2020-06-10 19:34:25 -05:00
asonix
9b45751f70 Add import API for filename preservation 2020-06-10 19:26:54 -05:00
asonix
1fbb5005ef Keep track of image variants rather than walking fs on delete 2020-06-09 18:40:01 -05:00
asonix
9a3b8dd4d7 Add GET for deleting images 2020-06-09 17:37:48 -05:00
Dessalines
0e7d4fc34f Upgrading rust-musl and alpine, adding filter env var. 2020-06-07 18:29:22 -04:00
Dessalines
15cf50e3b1 Merge remote-tracking branch 'asonix/master' 2020-06-07 16:21:29 -04:00
Dessalines
a7bc7c0121 Adding docker and docker-compose. 2020-06-07 16:21:25 -04:00
asonix
4f55251310 Add filter whitelist 2020-06-07 14:12:19 -05:00
asonix
35aca6a426 Unneeded mut 2020-06-07 13:03:36 -05:00
asonix
ab8a24f25e Identity doesn't alter path 2020-06-07 13:02:49 -05:00
asonix
fe0082a23b Add a concept of 'processors' to allow multiple image transformations 2020-06-07 12:51:45 -05:00
asonix
d71b86a81e Put image crate actions behind web::block 2020-06-07 11:34:44 -05:00
asonix
fc1ae4be49 Add endpoint for downloading remote images 2020-06-07 10:59:58 -05:00
asonix
1b526f4c1f Remove redundant code 2020-06-06 22:24:33 -05:00
asonix
b7a66c7863 Remove outdated TODO 2020-06-06 22:21:42 -05:00
asonix
18e7aa8d06 Enable compression 2020-06-06 21:14:31 -05:00
asonix
459db42a88 Add optional image format coersion 2020-06-06 20:44:26 -05:00
asonix
981e146696 Add StructOpt 2020-06-06 19:54:06 -05:00
asonix
fcf18ceca5 Background deletions 2020-06-06 19:33:29 -05:00
asonix
d60862a0d0 Implement delete tokens 2020-06-06 19:29:15 -05:00
asonix
7298e500cb Introduce alias system 2020-06-06 17:43:33 -05:00
asonix
65b83c6a06 Split into multiple files 2020-06-06 16:41:17 -05:00
asonix
6d7673dfcc A Simple duplicate-resolving image host 2020-06-06 16:02:26 -05:00