GStreamer Editing Services -------------------------- This is a list of features and goals for the GStreamer Editing Services. Some features are already implemented, and some others not. When the status is not specified, this means it is still not implemented but might be investigated. FUNDAMENTAL GOALS: 1) API must be easy to use for simple use-cases. Use and abuse convenience methods. 2) API must allow as many use-cases as possible, not just the simple ones. * Project file load/save support (GESFormatter) Status: Implemented, requires API addition for all use-cases. Problems: Timelines can be stored in many different formats, we need to ensure it is as easy/trivial as possible for users to load/save those timelines. Some timeline formats might have format-specific sources/objects/effects which need to be handled in certain ways and therefore provide their own classes. The object that can save/load GESTimeline are Formatters. Formatters can offer support for load-only/save-only formats. There must be a list of well-known GES classes that all formatters must be able to cope with. If a subclass of one of those classes is present in a timeline, the formatter will do its best to store a compatible information. A Formatter can ask a pre-render of classes that it doesn't understand (See Proxy section). Formatters can provide subclasses of well-known GES classes when filling in the timeline to offer format-specific features. * Grouping/Linking of Multiple TrackObjects Status: Implemented, but doesn't have public API for controlling the tracked objects or creating groups from TimelineObject(s) Problems: In order to make the usage of timelines at the Layer level as easy as possible, we must be able to group any TrackObject together as one TimelineObject. The base GESTimelineObject keeps a reference to all the GESTrackObjects it is controlling. It contains a mapping of the position of those track objects relatively to the timeline objects. TrackObjects will move and be modified synchronously with the TimelineObject, and vice-versa. TrackObjects can be 'unlocked' from the changes of its controlling TimelineObject. In this case, it will not move and be modified synchronously with the TimelineObject. * Selection support (Extension from Grouping/Linking) Problems: In order to make user-interface faster to write, we must have a way to create selections of user-selected TimelineObject(s) or TrackObject(s) to move them together. This should be able to do by creating a non-visible (maybe not even inserted in the layer?) TimelineObject. * Effects support Status: Partially Implemented, requires API addition for all use-cases. Problems: In order for users to apply multimedia effects in their timelines, we need an API to search, add and control those effects. We must be able to provide a list of effects available on the system at runtime. We must be able to configure effects through an API in GES without having to access the GstElements properties directly. We should also expose the GstElements contained in an effect so it is possible for people to control their properties as they wish. We must be able to implement and handle complex effects directly in GES. We must be able to configure effects through time -> Keyframes without duplicating code from GStreamer. * Source Material object Problems: Several TimelineSource for a same uri actually share a lot in common. That information will mostly come from GstDiscoverer, but could also contain extra information provided by 3rd party modules. Definition: Material: n, The substance or substances out of which a thing is or can be made. In order to avoid duplicating that information in every single TimelineSource, a 'Material' object needs to be made available. A Material object contains all the information which is independent of the usage of that material in a timeline. A Material object can specify the TimelineSource class to use in a Layer. * Proxy support Problems: A certain content might be impossible to edit on a certain setup due to many reasons (too complex to decode in realtime, not in digital format, not available locally, ...). In order to be able to store/export timelines to some formats, one might need to have to create a pre-render of some items of the timeline, while retaining as much information as possible. Content here is not limited to single materials, it could very well be a complex combination of materials/effects like a timeline or a collection of images. To solve this problem, we need a notion of ProxyMaterial. It is a subclass of Material and as such provides all the same features as Material. It should be made easy to create one from an existing TimelineSource (and it's associated Material(s)), with specifiable rendering settings and output location. The user should have the possibility to switch from Proxy materials to original (in order to use the lower resolution/quality/... version for the editing phase and the original material for final rendering phase). * Editing modes (Ripple/Roll/Slip/Slide) Problems: Most editing relies on heavy usage of 4 editing tools which editors will require. Ripple/Roll happen on edit points (between two clips) and Slip/Slide happen on a clip. The Ripple tool allows you to modify the beginning/end of a clip and move the neighbour accordingly. This will change the overall timeline duration. The Roll tool allows you to modify the position of an editing point between two clips without modifying the inpoint of the first clip nor the out-point of the second clip. This will not change the overall timeline duration. The Slip tool allows you to modify the in-point of a clip without modifying it's duration or position in the timeline. The Slide tool allows you to modify the position of a clip in a timeline without modifying it's duration or it's in-point, but will modify the out-point of the previous clip and in-point of the following clip so as not to modify the overall timeline duration. These tools can be used both on TimelineObjects and on TrackObjects, we need to make sure that changes are propagated properly. * Coherent handling of Content in different formats Problems: When mixing content in different format (Aspect-Ratio, Size, color depth, number of audio channels, ...), decisions need to be made on whether to conform the material to a common format or not, and on how to conform that material. Conforming the material here means bringing it to a common format. All the information regarding the contents we are handling are stored in the various GESMaterial. The target format is also known throught the caps of the various GESTracks involved. Those two information will allow us to make decisions on what course of action to take. By default content should be conformed to a good balance of speed and avoid loss of information. Ex: If mixing a 4:3 video and a 16:9 video with a target track aspect ratio of 4:3, we will make the width of the two videos be equal without distorting their respective aspect-ratios. * Media Asset Management integration (Track, Search, Browse, Push content) TBD * Templates Problem: In order to create as quickly as possible professional-looking timelines, we need to provide a way to create 'templates' which users can select and have an automatic timeline 'look' used. This will allow users to be able to quickly add their clips, set titles and have a timeline with a professional look. This is similar to the same feature that iMovie offers both on desktop and iOS. * Plugin system Problem: All of GES classes are made in such a way that creating new sources, effects, templates, formatters, etc... can be easily added either to the GES codebase itself or to applications. But in order to provide more features without depending on GES releases, limit those features to a single application, and in order to provide 'closed'/3rd party features, we need to implement a plugin system so one can add new features. Use a registry system similar to GStreamer.