Allow passing lambda functions to Nested

CHANGELOG.rst

@@ -1,12 +1,63 @@
 Changelog
 ---------
 
+3.3.0 (unreleased)
+******************
+
+Features:
+
+- ``fields.Nested`` may take a callable that returns a schema instance.
+  Use this to resolve order-of-declaration issues when schemas nest each other (:issue:`1146`).
+
+.. code-block:: python
+
+    # <3.3
+    class AlbumSchema(Schema):
+        title = fields.Str()
+        artist = fields.Nested("ArtistSchema", only=("name",))
+
+
+    class ArtistSchema(Schema):
+        name = fields.Str()
+        albums = fields.List(fields.Nested(AlbumSchema))
+
+
+    # >=3.3
+    class AlbumSchema(Schema):
+        title = fields.Str()
+        artist = fields.Nested(lambda: ArtistSchema(only=("name",)))
+
+
+    class ArtistSchema(Schema):
+        name = fields.Str()
+        albums = fields.List(fields.Nested(AlbumSchema))
+
+Deprecations:
+
+- Passing the string ``"self"`` to ``fields.Nested`` is deprecated.
+  Use a callable instead.
+
+.. code-block:: python
+
+    from marshmallow import Schema, fields
+
+    # <3.3
+    class PersonSchema(Schema):
+        partner = fields.Nested("self", exclude=("partner",))
+        friends = fields.List(fields.Nested("self"))
+
+
+    # >=3.3
+    class PersonSchema(Schema):
+        partner = fields.Nested(lambda: PersonSchema(exclude=("partner")))
+        friends = fields.List(fields.Nested(lambda: PersonSchema()))
+
 3.2.2 (2019-11-04)
 ******************
 
 Bug fixes:
 
-- Don't load fields for which ``load_only`` and ``dump_only`` are both ``True`` (:pr:`1448`). 
+- Don't load fields for which ``load_only`` and ``dump_only`` are both ``True`` (:pr:`1448`).
 - Fix types in ``marshmallow.validate`` (:pr:`1446`).
 
 Support:

docs/nesting.rst

@@ -22,7 +22,7 @@ Schemas can be nested to represent relationships between objects (e.g. foreign k
             self.title = title
             self.author = author  # A User object
 
-Use a :class:`Nested <marshmallow.fields.Nested>` field to represent the relationship, passing in a nested schema class.
+Use a :class:`Nested <marshmallow.fields.Nested>` field to represent the relationship, passing in a nested schema.
 
 .. code-block:: python
 
@@ -53,24 +53,24 @@ The serialized blog will have the nested user representation.
     #             'created_at': '2014-08-17T14:58:57.600623+00:00'}}
 
 .. note::
-    If the field is a collection of nested objects, you must set ``many=True``.
+    If the field is a collection of nested objects, pass the `Nested <marshmallow.fields.Nested>` field to `List <marshmallow.fields.List>`.
 
     .. code-block:: python
 
-        collaborators = fields.Nested(UserSchema, many=True)
+        collaborators = fields.List(fields.Nested(UserSchema))
 
 .. _specifying-nested-fields:
 
 Specifying Which Fields to Nest
 -------------------------------
 
-You can explicitly specify which attributes of the nested objects you want to serialize with the ``only`` argument.
+You can explicitly specify which attributes of the nested objects you want to (de)serialize with the ``only`` argument to the schema.
 
 .. code-block:: python
 
     class BlogSchema2(Schema):
         title = fields.String()
-        author = fields.Nested(UserSchema, only=["email"])
+        author = fields.Nested(UserSchema(only=("email",)))
 
 
     schema = BlogSchema2()
@@ -81,15 +81,15 @@ You can explicitly specify which attributes of the nested objects you want to se
     #     'author': {'email': u'[email protected]'}
     # }
 
-You can represent the attributes of deeply nested objects using dot delimiters.
+Dotted paths may be passed to ``only`` and ``exclude`` to specify nested attributes.
 
 .. code-block:: python
 
     class SiteSchema(Schema):
         blog = fields.Nested(BlogSchema2)
 
 
-    schema = SiteSchema(only=["blog.author.email"])
+    schema = SiteSchema(only=("blog.author.email",))
     result = schema.dump(site)
     pprint(result)
     # {
@@ -125,8 +125,6 @@ You can replace nested data with a single value (or flat list of values if ``man
     # }
 
 
-You can also exclude fields by passing in an ``exclude`` list. This argument also allows representing the attributes of deeply nested objects using dot delimiters.
-
 .. _partial-loading:
 
 Partial Loading
@@ -168,27 +166,29 @@ You can specify a subset of the fields to allow partial loading using dot delimi
 Two-way Nesting
 ---------------
 
-If you have two objects that nest each other, you can refer to a nested schema by its class name. This allows you to nest Schemas that have not yet been defined.
-
+If you have two objects that nest each other, you can pass a callable to `Nested <marshmallow.fields.Nested>`.
+This allows you to resolve order-of-declaration issues, such as when one schema nests a schema that is declared below it.
 
-For example, a representation of an ``Author`` model might include the books that have a foreign-key (many-to-one) relationship to it. Correspondingly, a representation of a ``Book`` will include its author representation.
+For example, a representation of an ``Author`` model might include the books that have a many-to-one relationship to it.
+Correspondingly, a representation of a ``Book`` will include its author representation.
 
 .. code-block:: python
 
-    class AuthorSchema(Schema):
-        # Make sure to use the 'only' or 'exclude' params
+    class BookSchema(Schema):
+        id = fields.Int(dump_only=True)
+        title = fields.Str()
+
+        # Make sure to use the 'only' or 'exclude'
         # to avoid infinite recursion
-        books = fields.Nested("BookSchema", many=True, exclude=("author",))
+        author = fields.Nested(lambda: AuthorSchema(only=("id", "title")))
 
-        class Meta:
-            fields = ("id", "name", "books")
 
+    class AuthorSchema(Schema):
+        id = fields.Int(dump_only=True)
+        title = fields.Str()
 
-    class BookSchema(Schema):
-        author = fields.Nested(AuthorSchema, only=("id", "name"))
+        books = fields.List(fields.Nested(BookSchema(exclude=("author",))))
 
-        class Meta:
-            fields = ("id", "title", "author")
 
 .. code-block:: python
 
@@ -221,27 +221,54 @@ For example, a representation of an ``Author`` model might include the books tha
     #   ]
     # }
 
+You can also pass a class name as a string to `Nested <marshmallow.fields.Nested>`.
+This is useful for avoiding circular imports when your schemas are located in different modules.
+
+.. code-block:: python
+
+    # books.py
+    from marshmallow import Schema, fields
+
+
+    class BookSchema(Schema):
+        id = fields.Int(dump_only=True)
+        title = fields.Str()
+
+        author = fields.Nested("AuthorSchema", only=("id", "title"))
+
+.. code-block:: python
+
+    # authors.py
+    from marshmallow import Schema, fields
+
+
+    class AuthorSchema(Schema):
+        id = fields.Int(dump_only=True)
+        title = fields.Str()
+
+        books = fields.List(fields.Nested("BookSchema", exclude=("author",)))
+
 .. note::
-    If you need to, you can also pass the full, module-qualified path to `fields.Nested`. ::
 
-        books = fields.Nested('path.to.BookSchema',
-                              many=True, exclude=('author', ))
+    If you have multiple schemas with the same class name, you must pass the full, module-qualified path. ::
+
+        author = fields.Nested("authors.BookSchema", only=("id", "title"))
 
 .. _self-nesting:
 
 Nesting A Schema Within Itself
 ------------------------------
 
-If the object to be marshalled has a relationship to an object of the same type, you can nest the `Schema` within itself by passing ``"self"`` (with quotes) to the :class:`Nested <marshmallow.fields.Nested>` constructor.
+If the object to be marshalled has a relationship to an object of the same type, you can nest the `Schema` within itself by passing a callable that returns an instance of the same schema.
 
 .. code-block:: python
 
     class UserSchema(Schema):
         name = fields.String()
         email = fields.Email()
-        friends = fields.Nested("self", many=True)
         # Use the 'exclude' argument to avoid infinite recursion
-        employer = fields.Nested("self", exclude=("employer",), default=None)
+        employer = fields.Nested(lambda: UserSchema(exclude=("employer",)))
+        friends = fields.List(fields.Nested(lambda: UserSchema()))
 
 
     user = User("Steve", "[email protected]")

docs/upgrading.rst

@@ -3,6 +3,55 @@ Upgrading to Newer Releases
 
 This section documents migration paths to new releases.
 
+Upgrading to 3.3
+++++++++++++++++
+
+In 3.3, `fields.Nested <marshmallow.fields.Nested>` may take a callable that returns a schema instance. 
+Use this to resolve order-of-declaration issues when schemas nest each other.
+
+.. code-block:: python
+
+    from marshmallow import Schema, fields
+
+    # <3.3
+    class AlbumSchema(Schema):
+        title = fields.Str()
+        artist = fields.Nested("ArtistSchema", only=("name",))
+
+
+    class ArtistSchema(Schema):
+        name = fields.Str()
+        albums = fields.List(fields.Nested(AlbumSchema))
+
+
+    # >=3.3
+    class AlbumSchema(Schema):
+        title = fields.Str()
+        artist = fields.Nested(lambda: ArtistSchema(only=("name",)))
+
+
+    class ArtistSchema(Schema):
+        name = fields.Str()
+        albums = fields.List(fields.Nested(AlbumSchema))
+
+A callable should also be used when nesting a schema within itself.
+Passing ``"self"`` is deprecated.
+
+.. code-block:: python
+
+    from marshmallow import Schema, fields
+
+    # <3.3
+    class PersonSchema(Schema):
+        partner = fields.Nested("self", exclude=("partner",))
+        friends = fields.List(fields.Nested("self"))
+
+
+    # >=3.3
+    class PersonSchema(Schema):
+        partner = fields.Nested(lambda: PersonSchema(exclude=("partner")))
+        friends = fields.List(fields.Nested(lambda: PersonSchema()))
+
 .. _upgrading_3_0:
 
 Upgrading to 3.0

docs/why.rst

@@ -33,8 +33,8 @@ As an example, you might have a JSON endpoint for retrieving all information abo
 
     class GameStateSchema(Schema):
         _id = fields.UUID(required=True)
-        players = fields.Nested(PlayerSchema, many=True)
         score = fields.Nested(ScoreSchema)
+        players = fields.List(fields.Nested(PlayerSchema))
         last_changed = fields.DateTime(format="rfc")
 
         class Meta:

examples/peewee_example.py

@@ -77,7 +77,7 @@ def wrap(self, data, many, **kwargs):
 class TodoSchema(Schema):
     id = fields.Int(dump_only=True)
     done = fields.Boolean(attribute="is_done", missing=False)
-    user = fields.Nested(UserSchema, exclude=("joined_on", "password"), dump_only=True)
+    user = fields.Nested(UserSchema(exclude=("joined_on", "password")), dump_only=True)
     content = fields.Str(required=True)
     posted_on = fields.DateTime(dump_only=True)
 

src/marshmallow/fields.py

@@ -437,10 +437,19 @@ class Nested(Field):
 
     Examples: ::
 
-        user = fields.Nested(UserSchema)
-        user2 = fields.Nested('UserSchema')  # Equivalent to above
-        collaborators = fields.Nested(UserSchema, many=True, only=('id',))
-        parent = fields.Nested('self')
+        class ChildSchema(Schema):
+            id = fields.Str()
+            name = fields.Str()
+            # Use lambda functions when you need two-way nesting or self-nesting
+            parent = fields.Nested(lambda: ParentSchema(only=("id",)), dump_only=True)
+            siblings = fields.List(fields.Nested(lambda: ChildSchema(only=("id", "name"))))
+
+        class ParentSchema(Schema):
+            id = fields.Str()
+            children = fields.List(
+                fields.Nested(ChildSchema(only=("id", "parent", "siblings")))
+            )
+            spouse = fields.Nested(lambda: ParentSchema(only=("id",)))
 
     When passing a `Schema <marshmallow.Schema>` instance as the first argument,
     the instance's ``exclude``, ``only``, and ``many`` attributes will be respected.
@@ -471,7 +480,7 @@ class Nested(Field):
 
     def __init__(
         self,
-        nested: typing.Union[SchemaABC, type, str],
+        nested: typing.Union[SchemaABC, type, str, typing.Callable[[], SchemaABC]],
         *,
         default: typing.Any = missing_,
         only: types.StrSequenceOrSet = None,
@@ -487,6 +496,12 @@ def __init__(
             raise StringNotCollectionError(
                 '"exclude" should be a collection of strings.'
             )
+        if nested == "self":
+            warnings.warn(
+                "Passing 'self' to `Nested` is deprecated. "
+                "Use `Nested(lambda: MySchema(...))` instead.",
+                DeprecationWarning,
+            )
         self.nested = nested
         self.only = only
         self.exclude = exclude
@@ -505,8 +520,13 @@ def schema(self):
         if not self._schema:
             # Inherit context from parent.
             context = getattr(self.parent, "context", {})
-            if isinstance(self.nested, SchemaABC):
-                self._schema = copy.copy(self.nested)
+            if callable(self.nested) and not isinstance(self.nested, type):
+                nested = self.nested()
+            else:
+                nested = self.nested
+
+            if isinstance(nested, SchemaABC):
+                self._schema = copy.copy(nested)
                 self._schema.context.update(context)
                 # Respect only and exclude passed from parent and re-initialize fields
                 set_class = self._schema.set_class
@@ -521,20 +541,17 @@ def schema(self):
                     self._schema.exclude = set_class(self.exclude).union(original)
                 self._schema._init_fields()
             else:
-                if isinstance(self.nested, type) and issubclass(self.nested, SchemaABC):
-                    schema_class = self.nested
-                elif not isinstance(self.nested, (str, bytes)):
+                if isinstance(nested, type) and issubclass(nested, SchemaABC):
+                    schema_class = nested
+                elif not isinstance(nested, (str, bytes)):
                     raise ValueError(
-                        "Nested fields must be passed a "
-                        "Schema, not {}.".format(self.nested.__class__)
+                        "`Nested` fields must be passed a "
+                        "`Schema`, not {}.".format(nested.__class__)
                     )
-                elif self.nested == "self":
-                    ret = self
-                    while not isinstance(ret, SchemaABC):
-                        ret = ret.parent
-                    schema_class = ret.__class__
+                elif nested == "self":
+                    schema_class = self.root.__class__
                 else:
-                    schema_class = class_registry.get_class(self.nested)
+                    schema_class = class_registry.get_class(nested)
                 self._schema = schema_class(
                     many=self.many,
                     only=self.only,