Incremental update

[dmpetrov]

Description

User can do incremental update manually like:

def my_embedding(file: File) -> list[float]:
    return [...]

dc = DataChain.from_storage("s3://bkt/dir1/*.jpg")
# Create 1st version
dc = dc.map(emd=my_embedding).save("image_emb")

...
# update

new = DataChain.from_storage("s3://bkt/dir1/*.jpg")
old = DataChain.from_dataset("image_emb")
diff = new.diff(old).map(emd=my_embedding)

# Create 2nd version
res = old.union(diff).save("image_emb")

It would be great if this can be supported out of the box. Users could then update datasets directly from the UI.

def my_embedding(file: File) -> list[float]:
    return [...]

# Create 1st version
dc = DataChain.incremental_dataset("s3://bkt/dir1/*.jpg", my_embedding, "image_emb")
...
# Update to 2st version
dc = dc.update()

Challenges:

• We need to preserve the environment since we use a custom python code my_embedding(). So, Inline project meta #776 might be a prerequisite.
• Not every dataset is updatable. Do we need a special type (hopefully not)?
• It might be better to generalize this story to updatable-pipelines and solve this, more general case.