UI Workflow Discussion - Timeline Clip Workflow

[BillyDM]

How should the workflow of arranging synths, audio clips, automation clips, and midi clips?

First, some terminology:

• "Vertical workflow" shall refer to the more "traditional" method of restricting a synth or "audio clip track" to a single row on the timeline. Automation clips appear in a row below the track it's automating, and these automation lanes can be shown/hidden. These lanes are a one-to-one correlation to the tracks in the mixer.
• "Pattern workflow" shall refer to the "FL Studio"-style method where synths and audio clips are not tied to a specific row on the timeline. Instead, synths and "audio clip tracks" are free to be assigned to any mixer track. Also midi clips, automation clips, and audio clips are free to be placed anywhere on the timeline grid.

For context, here are some pros and cons to each method:

Vertical Workflow

Pros

• There is a clear and easy to understand one-to-one correlation between the lanes on the timeline and the tracks on the mixer.
• It is more clear which midi, automation, and audio clips are assigned to which synth / "audio clip track".
• It is easier to copy-paste clips between different instruments.
• It is easier to group tracks/instruments together into "folders".
• The "track header" (the little panel that shows the track name, a fader to control volume, a db meter, arm/record/solo/mute buttons) is very standard and familiar.
• The one-to-one correlation makes organizing projects easier.

Cons

• It is more restrictive on how the user is allowed to organize their project.
• Hiding/showing/adding automation lanes for a particular track is very finicky.
• In a project with a lot of tracks and automation, a lot of vertical space will be taken up by lanes resulting in either a lot of scrolling or showing/hiding of lanes.
• Copying-pasting and looping/linking clips is more finicky.
• Extra work would need to go into making the whole "grouping tracks into folders" feature.
• Creating "layers" of instruments (using a single midi clip to control all the layers) is more clunky and requires special "routing plugins" on the track fx.
• Non-destructive loop recording will be more difficult (Do we add another tab to hide/show various takes on the track header? How does the user select which take takes precedent?)

Pattern Workflow

Pros

• It is a very "free-flow" way of working, allowing the user to place and organize clips in whatever manner they like.
• It is easy to select various patterns and copy-paste them wherever in the project, and to have these clips "linked" together so when one is edited it affects all the other instances in the timeline.
• In a project with a lot of clips, the user can organize all the clips on the timeline in a way that they are always visible, as opposed to the vertical workflow where a lot of vertical space would be taken up by each lane resulting in either a lot of scrolling or showing/hiding of lanes.
• It it easier to "layer" instruments together so you can use a single midi clip to control all synth layers for example.
• It it easier to route multiple synths and audio clips into the same mixer track.
• Non-destructive loop recording is easier (just add the recording clip on the next available timeline row).

Cons

• It is not as obvious and intuitive for users who are completely new to DAWs.
• There is no clear correlation between the synths / audio clips and the mixer track they are assigned to.
• There is no clear correlation between the midi / automation clips and the synth / "audio clip track" they are assigned to.
• It can be confusing for the user to keep track of all the patterns and clips in the project, and can obviously lead to very messy projects if the user doesn't take the time to organize.
• We would need an additional view for a sort-of "channel rack". We don't have to do this exactly like FL does though.
• We would also need some way to distinguish when the user wanted to open the piano roll/automation editor, and when they want to open the instrument plugin that the clip is assigned to.
• It can be more finicky to distinguish between when the user wants to copy-paste a "linked" clip or a new unique "unlinked" clip.

It is clear there is a split of workflow preference here. This issue is meant to be the place of discussion on how to solve the issue of satisfying both workflow tastes.

Please post any ideas and design mockups of those ideas here :)

[DGriffin91]

There's an idea that's been discussed a bit in the discord that allows for a possible hybrid approach. The general idea is similar to compound clips, a concept often found in video editors (not really sure if this is the best name). These would be clips that can contain one or more sub clips and their associated tracks. The contained clips could be both midi and audio clips. Compound clips could be placed on any track in the DAW and their audio & midi output would move through the track hierarchy like any other clip.

In the case where it's just a single midi clip with a synth, it would be really simple. But it could scale up, where you take a whole folder of tracks and make it into a compound clip. Copies of that compound clip would be instances and an edit of any of them would propagate to all of them. The compound clip could also be linked from an external project, where the whole other project file is the compound clip. Then edits to that other file can result in updates in the current file. (like linking in blender) This can also provide a potentially more basic form of collaboration, where the linked compound clips could not just be a file on disk but also something that is synchronized over the network.

I made a bit of a mockup of this idea. It's pretty rough and needs refining, but shows a bit of what it could look like.
Group track before making it into a compound clip:

Then with some menu thing it's made into compound clips that can be instanced around:

And they can be edited in place:

When editing in place it would show the sub clips in the compound clip offset to the parent composite clip that is edited.
In the case where this is only used to attach a synth to some midi data, the midi item could still be accessed directly, it would not have to be expanded. This is just a larger example where an entire group/folder track is used.
The FX rack on the side could also update to show the FX rack for the compound clip when it is selected without having to open it.

Another thing the compound clip could possibly do, if it's input is only time (there aren't any sends into the group or other variants) the output of the whole composite clip could be cached. This would make the multiple instances have a minimal performance impact, and it could even be sliced up, reversed, etc... non destructively like a wav file. Then any of the slice references or whole composite instances could be opened up and edited, after which the output is cached again.

This output cache could also be used in cases where multiple people are working on the same project, but not everyone has all of the same source material or plugins. If someone had an remote instance of someone else's composite clip, and they were missing some plugins, they could still play it back because the collaboration system could send over the rendered cache. They just wouldn't be able to open the composite clip to edit it.

[geom3trik]

Suggestion for a change in terminology. Correct me if I'm wrong but the essence of the options seems to come down to whether it's the track that has the information about the instrument/source, or whether it's the clip (pattern). Therefore I propose that we refer to the options as 'track-based' (Ableton, Bitwig) or 'clip-based' (FL Studio), when comparing.

[ollpu]

Here's an idea (inspired by @DGriffin91 on voice) that solely addresses vertical space in the "vertical workflow".

A group/folder might look something like this:

When collapsed, the clips contained in its subtracks are be visible in a similar way as they were before, assuming there's only one clip active at a time. The clips can still be selected, manipulated and opened for editing like normal. New clips cannot be added, however.

For overlapping clips, they have to be laid out differently, e.g. as shown on the right. I've taken out the piano roll / waveform previews, but for small amounts of tracks they could still be shown. These clips can also be manipulated directly.

The same idea could apply to the automation hierarchy.

[ANVF-Postal]

Postal's Timeline Proposal

After talking with @DGriffin91 about the workflows of FL and Reaper, I really got a chance to see the areas where FL lacks both capabilities and convenience. With a bit of back and forth, I've mocked up a full scale concept of what I believe is an ideal hybrid approach. Here it is.

The goal of this mockup was to implement the storage, flexibility and less destructive aspects of a pattern (or clip) system, while maintaining the intuition and power of a track based system. With this workflow, the view of the channel rack all but mirrors the view of the mixer, timelines are flexible and unlocked, midi can be easily copied between instruments by drag-and-drop, clips are visibly and automatically distinguished as being a part of one instrument or group, and the clips (or patterns) on each rack track can be collapsed so they'll never take up fixed space on your screen.

A brief highlight of the mockup's parts:
-A drum sequencer, with a pattern being edited currently.
-A single instrument pattern below the group, where it's only playing (and is only able to play) the kick sample.
-Multiple instruments on one timeline track
-Vital midi clip (pattern) on the timeline that isn't visible on the rack, since it's collapsed.
-Volume and pan knobs, per-rack track
-Two automation clips, stored in the same way as a MIDI pattern.
-Dots on the left of each rack track indicate that they can be reordered and parented, just by clicking and dragging.
-In the sequencer, the pattern name box has arrows next to it, indicating that scrolling over or clicking the button will cycle to the next pattern. Notice how pattern 1 is selected in the rack with an outline below the drum group.
-Patterns can be renamed away from the standard naming scheme. Every Midi clip/pattern is named %SYNTH_NAME% - Pattern %NUMBER% by default. E.g Vital - Pattern 1.
-The Kick has a "Waveform" pattern. This an audio clip, for dragging into the timeline. Better for crashes where you want to see the duration on the timeline.
-There are three types of clips. Midi clips (patterns) which can be dragged to other synths and copied. Automation clips, which could be copied the same way (though remapping controls is hard) and audio clips, which are only for samples and can only be dragged to the timeline.

Something currently not shown in the mockup is the mixer.
The order of the rack tracks is meant to very closely mirror their appearance on the mixer. Groups will show up as folder tracks on the mixer (see bitwig) and all child rack tracks will go through the parent rack's effects on the mixer, ideal for drum bussing. The mixer is where you'd actually side-chain things, and do more extensive routing, but from the moment a new track is put in the channel rack, it's instantly assigned to its own mixer slot. This does a lot of background-organizing for the user, a feature not present in FL, where every channel must be manually assigned a mixer track. Of course in FL, this is meant to allow multiple instruments on the same track, but this is an inferior workflow to separate instruments having their own mixer tracks and being routed to a group/folder track.

Another feature not seen is the way midi notes would show up in a grouped sequencer. The sequencer tiles would essentially switch to a preview of the midi clip, the same as a pattern (midi) clip shown on the timeline.

What if a pattern is too long to fit in the width of the channel rack?
The rack could obviously extended by the user, giving more space for display. However in more compact views, the pattern should be scrolled left and right, depending on the user's X (maybe even Y) coordinate on the pattern clip's box.

What's the actual workflow?

On a new project, the rack is empty. Only a '+' button is visible in the channel rack, where the user can select what they'd like to add. Alternatively, synths and samples can be dragged and dropped in from a content browser.
Once a plugin is added to the rack, it's assigned a mixer track, and maybe (optionally) given a color.
A '+' button is visible below the instrument, indicating that clicking it will add a new blank Midi clip (AKA pattern)
Once clicked, the user gets brought to the piano roll (if it's a single instrument or a child) where they can place notes.
After the notes are placed, the piano roll can be closed, and the pattern dragged onto the timeline.

Easy!

What else?

I have a small document of notes outlining the smaller details of the system, but I don't think it's in the scope of this post.
I couldn't identify any fundamental problems with this system, and I believe the hierarchical design makes it suitable for programming, as well as endless expansion and organization within one project.

Since this is a mockup, I welcome any criticism and suggestions for how to improve it, as well as features to add that I might have overlooked.

Happy commenting

[awallenfang]

I really like this approach, since it fixes my main problem with FL's workflow, which has always just been the initial disconnect between the instruments/audio clips and the mixer.
I'm assuming that the pan and volume knobs are connected to the mixer's pan and volume, unlike FL, which is also a really nice improvement, allowing you to do simple mixing jobs without having to open up the whole mixer.

I feel like a responsive visual element, like a light that reacts to the sound and turns red when the track clips, could be added to make the direct connection to the mixer more apparent to new users, as well as give quick feedback for clipping tracks.

[BillyDM]

BillyDM's Timeline Proposal

Improving on Postal's idea, here is my idea:

In this workflow, the "Instruments"/"Busses" column is essentially a one-to-one representation of the mixer, but just vertical. These tracks can be grouped like in Bitwig. The "Busses" section can be shown/hidden with the little arrow (it will be hidden by default).

The column to the right of the "Instruments"/"Busses" column lists all the items that are "attached" to the selected track. From left to right the tabs show the vertical FX rack, piano roll clips, audio clips, and then automation clips. This way everything that can modify a particular track is grouped into one place.

The user can add instruments and audio clips to a new track by either dragging and dropping from the browser on the left side, or they can select the "+" button on the bottom of the instruments to add an empty track.

Likewise, the user can also add effects, midi clips, audio clips, and automation clips either by dragging and dropping them from the browser on the left, or by selecting the "+" button.

In the timeline, the user can place clips on whatever "lane" they wish like FL (I think it will be important to distinguish between what is a "lane" and what is a "track"). When a clip is single-clicked, it also highlights the track that it is assigned to. When a clip is double-clicked, it opens the respective editor (piano roll, audio clip editor, automation editor).

I think this has a sort of logical "flow" from left to right. You go from the browser, to the tracks, to the fx/clips, then to the timeline.

[PORTALSURFER]

Ideas from discord.
Considering compound/channel-clips, and channels

Clips in this idea would house midi/audio/automation data, and pipe it to channels.
The channels would house things like vst instruments, hardware synths, fx, etc.
These clips can live anywhere in the arranger. And the arranger lanes would be visual only for the most part, they would not affect routing. Volume, muting, solo, still makes sense here I think.

Solo/Mute would also work based on clip selection, to easily isolate the sounds you want to edit.

So in this workflow, you start by making some clips, and make some nice music.

You like what this section of music sounds like and want to reuse it.
So you group them into a channel/compound clip.
Now, there are 2 ways to do this. Note, these would not be instanced.

One way will simply group the clips into a masterclip, nothing changes in terms of routing here. Easy for reuse, clean, and no extra resources needed.

The second way is a unique grouping. This would duplicate all the channels, with its vsts, effects, etc, as well. Useful if you want to store what you made, decoupled, so you can keep messing around with the original channels.

You can then use these clips like any other clip, duplicating them around the timeline, cut them up, loop them, etc.

To edit these clips, you can select the channel/compound clip, and enter 'edit mode', perhaps by hitting TAB on your keyboard.
This would visually expand the group clip, showing all the sub-clips as if regular audio/note/automation clips, allowing you to select them, edit them, etc.
When done, you can collapse the group clip again by selecting the master and again hitting TAB.

[PORTALSURFER]

Ideas from discord part 2
Clip based automation, with crossfades

The idea here is to have instruments remember their base state, the defaults for each parameter.
And have automation override these defaults only when active, then snap back to the default state, if so desired.

This tries to solve the problem where one spends hours adjusting automation for a clip, only to adjust the automation elsewhere in another clip, which affects the previous clip because of the way automation usually works, and lose all the hard work.
Sometimes you want automation to remember the last value, but often you also do not. This should be controlled by the user.

automation nodes could have a binary state, hold or release, easily toggled between by the user.
hold, would keep the parameter at the last value until it is overwritten by new automation data (common for most daws)
release, would snap back to the default state, which can be stored in the instrument preset by the user.

In addition it could be possible to add crossfades in between clip automations, not unlike audio crossfades.
This would help prevent artifacts from very quick value changes for example, giving more control and a more natural feel.

[PORTALSURFER]

I really like the idea of selection something, and pressing TAB to jump into edit mode for that thing
This is clean, consistent, and helps with real estate by hiding the editors when not required.

The idea here is to select what you want to edit, and hit TAB to enter and exit edit mode.

if it is a note clip you go into a piano roll editor
if it is a compound clip you expand the subclips
if you have automation selected you jump into an automation editing mode

Hitting tab again will either push you deeper, like for example after selecting a subclip inside of a compound clip.
Or will pull you out of edit mode.
ctrl-TAB could 'escape', always forcing you back out of everything back to the root mode.

[PORTALSURFER]

Workflow idea for channel based system.

I was wondering how it would work when making new clips. In the more common 'track' based workflow you set up your routing chain, and then just draw clips on that track. With our far more free clip based routing idea this won't work.

So, I had the idea, what if you can drag/drop channels from the left onto the arranger to quickly create a clip which is routed to said channel. Could even work with multiple channels selected, and/or combined with a paint mode, where you select channels, then paint clips into the arranger.

[BillyDM]

So, I had the idea, what if you can drag/drop channels from the left onto the arranger to quickly create a clip which is routed to said channel.

Yeah, I think that's a good idea.