Flesh out remaing feature descriptions, fixes, changes, add upgrade notes

2024-05-14 01:32:42 +00:00 · 2023-07-08 21:34:21 -05:00 · 2023-07-08 21:34:21 -05:00 · fa74e3c82d
parent 535ee33a97
commit fa74e3c82d
1 changed files with 376 additions and 5 deletions
--- a/releases/0.4.0.md
+++ b/releases/0.4.0.md
@ -1,8 +1,8 @@
 # pict-rs 0.4.0

 I am excited to announce a new stable version of pict-rs, version 0.4.0. There's a few big things in
-0.4 that might be exciting to developers and admins. Below you'll find a list of changes from 0.3 to
-0.4.
+0.4 that might be exciting to developers and admins. Below you'll find upgrade notes for
+administrators, as well as a list of changes from 0.3 to 0.4.

 ## Overview

@ -15,6 +15,10 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's
 - [Full Video](#full-video)


+### Changes
+
+- [Default video codec](#default-video-codec)
+
 ### Other Features:

 - [Published Dependencies](#published-dependencies)
@ -24,10 +28,12 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's
 - [GIF configuration](#gif-configuration)
 - [JXL and AVIF](#jxl-and-avif)
 - [H265, VP8, VP9, and AV1](#h265-vp8-vp9-and-av1)
+- [Audio Codecs](#audio-codecs)
 - [404 Image](#404-image)
 - [DB Export](#db-export)
- [Variant cleanup endpoint](#variant-cleanup-endpoint)
- [Client pool configuration](#client-pool-configuration)
+- [Identifier Endpoint](#identifier-endpoint)
+- [Variant Cleanup Endpoint](#variant-cleanup-endpoint)
+- [Client Pool Configuration](#client-pool-configuration)


 ### Fixes
@ -42,11 +48,22 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's

 - [Purge by disk name](#purge-by-disk-name)
 - [Aliases by disk name](#aliases-by-disk-name)
+- [Filename endpoint](#filename-endpoint)


 ## Upgrade Notes

-TODO
+If you're running the provided docker container with default configuration values, the 0.4 container
+can be pulled and launched with no changes. There is a migration from the 0.3 database format to the
+0.4 database format that will occur automatically on first launch. After the migration is complete,
+the server will start.
+
+If you have any custom configuration, or are running outside of docker, see [Reworked
+Configuration](#reworked-configuration) below. It is likely that you will need to change
+configuration values for the migration to run properly.
+
+More information can be found in the [pict-rs
+readme](https://git.asonix.dog/asonix/pict-rs#user-content-0-3-to-0-4-migration-guide)


 ## Descriptions
@ -66,6 +83,38 @@ Notable changes:
 - `PICTRS__PATH` has become `PICTRS__OLD_DB__PATH`, `PICTRS__REPO__PATH` and
    `PICTRS__STORE__PATH`

+The pict-rs.toml equivalents:
+```toml
+# pict-rs.toml
+
+[server]
+# same format as the previous PICTRS__ADDRESS
+address = "0.0.0.0:8080"
+
+[tracing.logging]
+# same format as RUST_LOG
+targets = "info"
+
+[tracing.opentelemetry]
+# same format as RUST_LOG
+targets = "debug"
+
+[old_db]
+# same as the previous PICTRS__PATH
+path = "/mnt"
+
+[repo]
+# Added the `sled-repo` subdirectory to the previous PICTRS__PATH
+path = "/mnt/sled-repo"
+
+[store]
+# Added the `files` subdirectory to the previous PICTRS__PATH
+path = "/mnt/files"
+```
+
+As mentioned above, every configuration option has moved. Check the linked example configuration
+file for more information.
+

 ### Reworked Commandline

@ -178,6 +227,13 @@ It is important to note that while pict-rs is capable of storing videos, it's pr
 functionality is limited to just the video's thumbnail.


+### Default Video Codec
+
+pict-rs 0.4 has changed the default video codec from h264 to vp9. This was done because mp4 is still
+patent-encumbered, and I don't want to ship software the defaults to using it. This change may
+result is slower transcoding times.
+
+
 ### Published Dependencies

 pict-rs 0.3's experimental object storage support relied on a custom fork of the `rust-s3` library,
@ -192,3 +248,318 @@ enabling object storage. This library takes a sans-io approach, and was simple t


 ### HEAD Endpoints
+
+For the `/image/original/{filename}` and `/image/process.{ext}?src={filename}` endpoints, there are
+now analogous `HEAD` endpoints for fetching just the headers their `GET` counterparts would return.
+
+
+### Healthcheck Endpoints
+
+pict-rs now provides a `/healthz` endpoint that can be used to check if the running process is
+healthy. It currently attempts to write to the sled database and fetch information from the
+configured store, returning 200 if all operations succeeded.
+
+
+### Image Preprocessing
+
+pict-rs 0.4 introduces a new configuration option: `PICTRS__MEDIA__PREPROCESS_STEPS`. This option
+accepts the same syntax as the `process` endpoint, and can be used to apply transformations to all
+uploaded images.
+
+Here are some examples:
+
+```toml
+# pict-rs.toml
+
+[media]
+# Crop all uploaded media to a 16x9 aspect ratio, and then shrink to fit within a 1200 by 1200 pixel box.
+preprocess_steps = "crop=16x9&resize=1200"
+```
+
+```yaml
+# docker-compose.yml
+
+services:
+  pictrs:
+    image: asonix/pictrs:0.4.0
+    ports:
+      - "127.0.0.1:8080:8080"
+    restart: always
+    environment:
+      # Blur all uploaded images with a sigma value of 1.5
+      - PICTRS__MEDIA__PREPROCESS_STEPS=blur=1.5
+    volumes:
+      - ./volumes/pictrs:/mnt
+```
+
+
+### GIF configuration
+
+pict-rs 0.3 would transcode all uploaded GIF files to video formats. This is no longer the case.
+Some new configuration options have been introduced to determine when GIF files should and should
+not be converted to silent videos. By default, GIF files will remain GIFs if their dimensions are
+smaller than 128x128 px, and they don't have more than 100 frames. GIFs to that do fit these
+restrictions will be transcoded to a video format.
+
+To tweak these values, the following environment variables can be set:
+```yaml
+# docker-compose.yml
+
+services:
+  pictrs:
+    # ...
+    environment:
+      - PICTRS__MEDIA__GIF__MAX_WIDTH=128
+      - PICTRS__MEDIA__GIF__MAX_HEIGHT=128
+      - PICTRS__MEDIA__GIF__MAX_AREA=16384
+      - PICTRS__MEDIA__GIF__MAX_FRAME_COUNT=100
+```
+
+Alternatively, these values can be set in pict-rs.toml
+```toml
+# pict-rs.toml
+
+[media.gif]
+max_width = 128
+max_height = 128
+max_area = 16384
+max_frame_count = 100
+```
+
+
+### JXL and AVIF
+
+pict-rs 0.4 introduces support for two new image formts: JPEG XL (JXL) and AVIF. These two image
+formats offer better compression to quality ratios than other standard formats, although they may be
+less well-supported by browsers or other clients. Not only can these formats be uploaded and handled
+now, but they can also be used with the `PICTRS__MEDIA__FORMAT` variable for transcoding all
+uploaded images.
+
+Exmaples:
+```yaml
+# docker-compose.yml
+
+services:
+  pictrs:
+    # ...
+    environment:
+      - PICTRS__MEDIA__FORMAT=avif
+```
+
+```toml
+# pict-rs.toml
+[media]
+format = "jxl"
+```
+
+
+### H265, VP8, VP9, and AV1
+
+As mentioned earlier, the default video codec has changed from `h264` to `vp9`, however, pict-rs 0.4
+introduces support for a broader range of video codecs:
+- in the mp4 container
+  - h264
+  - h265
+- in the webm container
+  - vp8
+  - vp9
+  - av1
+
+The default video codec can be changed with the following settings:
+```yaml
+# docker-compose.yml
+
+services:
+  pictrs:
+    # ...
+    environment:
+      - PICTRS__MEDIA__VIDEO_CODEC=av1
+```
+
+```toml
+# pict-rs.toml
+
+[media]
+video_codec = "h265"
+```
+
+Note that h264 and h265 in the mp4 container are still patent-encumbered, and you may wish to avoid
+them.
+
+
+### Audio Codecs
+
+With the introduction of full video, pict-rs adds the ability to configure the audio codec used for
+uploaded videos. By default, pict-rs will choose a reasonable codec for the provided video codec.
+This means that there is no need to configure the audio codec yourself. The option is still
+available, though.
+
+Here is the existing mapping between video and audio codecs:
+- av1, vp8, and vp9 use opus by default
+- h264 and h265 use aac by default
+
+Available codecs for configuration are
+- opus
+- vorbis
+- aac
+
+Setting a preferred codec can be done with the following configurations:
+```yaml
+# docker-compose.yml
+
+services:
+  pictrs:
+    # ...
+    environment:
+      - PICTRS__MEDIA__AUDIO_CODEC=vorbis
+```
+
+```toml
+# pict-rs.toml
+
+[media]
+audio_codec = "opus"
+```
+
+
+### 404 Image
+
+pict-rs 0.4 adds support for setting a "Not Found" image. This image will be returned for any
+request for an image that does not exist on the server. This is done through an internal endpoint,
+which means it requires pict-rs' `PICTRS__SERVER__API_KEY` to be set.
+
+After uploading an image to pict-rs, it's name can be submitted to the `/internal/set_not_found`
+endpoint with the following JSON:
+```json
+{
+    "alias": "79087a4f-ea31-4878-9e8a-0498dada597b.webp"
+}
+```
+
+If the alias exists on the server, it will mark that image for use as the "404 image"
+
+
+### DB Export
+
+Part of administering a server means taking regular backups. Many backup systems rely on simply
+copying files around, which doesn't work well for pict-rs' sled database, which may change while
+being read by the backup tool.
+
+One solution to this is to use a snapshotting filesystem like btrfs, bcachefs, or zfs. A filesystem
+snapshot is "atomic", and will represent the database at an exact moment in time.
+
+For admins running on more common filesystems such as ext4 or xfs, there is now a configuration
+option and internal endpoint for producing immutable copies of the current database state.
+
+```toml
+# pict-rs.toml
+
+[repo]
+export_path = "./data/exports"
+```
+
+```yaml
+# docker-compose.yml
+
+services:
+  pictrs:
+    # ...
+    environment:
+      - PICTRS__REPO__EXPORT_PATH=/opt/pict-rs/exports
+```
+
+When this value is set, the `/internal/export` endpoint becomes available. Hitting that endpoint
+will result in pict-rs exporting its current state to a new timestamped directory inside the
+`export_path`. This directory contains the same structure as the `sled-repo` directory, and in the
+event of data loss, can be restored by simply moving it into place.
+
+Here's an example process for restoring from an export:
+1. Stop pict-rs
+2. Move your current `sled-repo` directory to a safe location (e.g. `sled-repo.bak`)
+    ```bash
+    $ mv sled-repo sled-repo.bak
+    ```
+3. Copy an exported database to `sled-repo`
+    ```bash
+    $ cp -r exports/2023-07-08T22:26:21.194126713Z sled-repo
+    ```
+4. Start pict-rs
+
+
+### Identifier Endpoint
+
+pict-rs 0.4 introduces a new endpoint for retrieving an alias' "True Name", the Identifier. An
+Identifier represents the real path to the file in filesystem storage or in object storage.
+Currently these paths are generated using the same algorithm, but pict-rs retains the right to
+optimize identifiers differently for filesystem and object storage in the future.
+
+
+### Variant Cleanup Endpoint
+
+This is one of the less exciting features, and won't be very commonly used. A new endpoint at
+`/internal/variants` accepts the DELETE verb. This will spawn a background task that iterates over
+all files in the database, removing any generated variants of an image, keeping only the Original.
+All variants can be generated again on request.
+
+
+### Client Pool Configuration
+
+Another less exciting feature, the number of connections pict-rs' internal HTTP Client maintains in
+its pool can now be configured. This value is 100 by default, and shouldn't need to be tweaked. If
+you are experiencing heavy traffic and are using object storage, you could increase the value. If
+you are constrained on allowed open file descriptors, you can decrease the value.
+
+Note that this value is multiplied by the number of threads pict-rs spawns, which depends on the
+number of cores allowed to the application. Limiting pict-rs to 2 CPUs with default pool settings
+will result in an effective limit of 200 connections.
+
+
+### GIF/webm Transparency
+
+In 0.3, pict-rs removed transparency from uploaded gifs and videos. 0.4 supports preserving the
+transparency of GIF and webm files on upload.
+
+
+### Image Orientation
+
+In 0.3, pict-rs would strip metadata from all images. This would remove metadata about the intended
+orientation of an image, and lead to uploaded media being sideways or upside down. pict-rs 0.4 still
+strips all metadata from uploaded images, but it now applies a rotation to images that require it as
+well. Images uploaded in version 0.4 should no longer be sideways or upside down.
+
+
+### io_uring File Write
+
+Due to a conflict between the `AsyncReadExt` trait and `StreamReader`, actix-web's `Payload` stream
+would be polled after completion when pict-rs' `io-uring` feature was enabled. This has been fixed
+by using the `read_buf` API rather than the `read_to_end` API in the io_uring file implementation.
+
+Note that io_uring is still considered an experimental feature, and probably should not be used in
+production. YMMV.
+
+
+### Blur Sigma
+
+pict-rs' `blur` filter would incorrectly apply the provided `sigma` value as the `radius` for a
+gaussian blur. This has been fixed in 0.4 to properly set the `sigma` value with a default `radius`
+of 0. This tells imagemagick to calculate a reasonable radius for the given image and sigma.
+
+
+### Purge by disk name
+
+Due to changes in how pict-rs stores files in 0.4 (introducing a pluggable store backend), the
+`/internal/purge` endpoint can no longer be provided with a `filename` query, and must be given an
+`alias` query instead.
+
+
+### Aliases by disk name
+
+Due to changes in how pict-rs stores files in 0.4 (introducing a pluggable store backend), the
+`/internal/aliases` endpoint can no longer be provided with a `filename` query, and must be given an
+`alias` query instead.
+
+
+### Filename Endpoint
+
+This was replaced by the Identifier endpoint, since files are no longer required to be stored on the
+filesystem.