@gabesullice @courajs thanks for reviewing!

My first reaction is that the "identities" key could use a little more elaboration. I'm least clear about its intended purpose.

The identities key was introduced in #1244 as a means to ensure that every lid that is passed in a request can be aligned with an id in the response, regardless of the shape of that response. This concept was born out of thinking through some exceptions in #1197 in which a response might take a shape that doesn't include all the created resources. However, as you point out, it's not strictly necessary with operations. I'm going to remove the concept from both #1244 and this PR to keep them both as lean as possible. Thanks for questioning this.

Ok, the identities array has been removed from both here and #1244, and the other suggestions / corrections have been incorporated.

One aspect that perhaps should be fleshed out further with examples is the usage of params in operations. I am reluctant to provide completely redundant examples with the query param examples, but I think some examples could be clarifying.

has http://jsonpatch.com/ also been considered as a basis? Everything here looks great and jsonpatch.com would give just a slight different envelope. An additional benefit of this would be the support of other, non-JSONAPI calls.

one minor other thing, "op" could potentially retain a more HTTP-style naming: GET, POST, PATCH, DELETE.

It's great to finally see this area get momentum @dgeb ❤️ ! I strongly agree with your bullet-points making the case that this should be part of the base spec - the scenarios are too common to overlook. I also think this proposal is more cohesive than what I've seen in the past; very much appreciate the work you have put in here.

I do have two main concerns:

• A. We're introducing a completely new payload, but all these concepts can be expressed with minor additions to the existing payload.
• B. Rather than allowing the client to specify the order of operations, it requires the client to do so.

Based on the PR description, I think we agree on A except for order of operations. Is this the case? Here's the "sideposting" doppleganger that would add/remove/update comments from an article, using op and lid keys from this PR:

PATCH /articles/1

{
  data: {
    id: "1" 
    type: "articles",
    relationships: {
      comments: {
        data: [
          { type: "comments", lid: "a", op: "add" },
          { type: "comments", id: 1, op: "remove" },
          { type: "comments", id: 2, op: "update" }
        ] 
      }
    }
  },
  included: [
    {
      type: "comments",
      lid: "a",
      attributes: { text: "I love JSONAPI!" }
    },
    {
      type: "comments",
      id: "2",
      attributes: { text: "An updated comment" }
    }
  ]
}

Note that this also supports creating 2 articles at the same time through sideposting to /bulk.

The benefits of a symmetrical payload seem so great to me, point B above (ordering operations) must be overwhelming if we are to introduce a new payload. But I'm not sure that's the case.

Consider a SQL database - you would, 99% of the time, want to persist the parent (e.g. article) before persisting the child (e.g. comments) (since we need the parent id before we can populate the foreign key). In these cases, forcing the client to specify order of operations seems nothing more than cruft (though I'm certainly open to allowing order of operations for the 1% use case).

There's a more important reason than cruft, though. Let's say I coded all my clients to persist the parent before the children while using a SQL database, then switched to a NoSQL store where this logic no longer applies. I would now want to persist these entities differently - but all my clients have already coded the logic to first persist the parent, then the child. I fear there are a number of examples like this, largely summarized by the statement:

Order of operations is a server implementation detail that is not the responsibility of the client.

Which is similar to the concern and examples raised by @gabesullice here.

I'd love to see order of operations supported optionally instead of required, both because it is almost always not needed and often seems not desirable. One way to support order of operations optionally is to take the above sideposting payload and add another key:

...
relationships: {
  comments: {
    data: [
      { type: "comments", lid: "a", op: "add", order: 1 }
      { type: "comments", lid: "b", op: "add", order: 2 }
  }
}
...

We'd end up mirroring the existing sideloading spec, with the addition of three total keys - lid and op, which are in this PR as well, and an optional order (which could even be an extension rather than base spec at that point).

I have some additional comments on the specifics of this PR but would love to get your thoughts on the above first. Again, thanks for the work here, it is awesome to see JSONAPI attempting to address this problem area.

has http://jsonpatch.com/ also been considered as a basis? Everything here looks great and jsonpatch.com would give just a slight different envelope. An additional benefit of this would be the support of other, non-JSONAPI calls.

@remmeier yes, this proposal was strongly inspired by JSON Patch. One significant difference is that update is used instead of replace to be consistent with the PATCH (instead of PUT) semantics. Another difference is that ref is used instead of a path pointer to acknowledge that the spec is targeting resources and resource collections, instead of a singular JSON document. Given these differences, I don't feel it's right to try to shoehorn exact compliance with RFC 6902 into this proposal, but it's definitely appropriate to call it out as an inspiration :)

one minor other thing, "op" could potentially retain a more HTTP-style naming: GET, POST, PATCH, DELETE.

Just like in JSON Patch, operations occur at a layer distinct from HTTP, and I didn't want to muddy the waters by pretending otherwise.

@gabesullice @richmolj thanks for considering this carefully and for your feedback!

There are a few aspects of this proposal about ordering and complexity that I'd like to address:

• Every operation has complete fidelity with a singular request. Everything possible with singular requests can be completed with operations, and vice versa. The only additional complexity is the usage of lids to bind together inputs and outputs if necessary. And the only additional requirement is for servers to process all the operations in a request transactionally.

• There is no additional knowledge required by clients of how a server persists resources beyond what is required by the existing spec. For instance, a relationship can't be created between an existing resource and one that has not yet been created yet. If request A can follow request B, then operation A can follow operation B.

• If order is not important, then I would strongly recommend making multiple singular requests in parallel. By using operations, the client is saying that order is important.

• If order can be elided by the server because of some knowledge it has, then I think it is free to do so. However, there should be no "evidence" in the response, because that breaks the contract with the client. If this allowance requires some tweaking of the language used in this proposal, I'm open to it.

Note: edited for clarity, but then reverted edits after I realized others had commented based on the original text.

thanks for considering this carefully and for your feedback!

Thanks for taking the time to address each concern so thoroughly!

Everything possible with singular requests can be completed with operations, and vice versa... There is no additional knowledge required by clients of how a server persists resources beyond what is required by the existing spec.

Thanks for clarifying this @dgeb, that's a subtlety that I didn't pick up on.

Perhaps this is something that could be stated explicitly in the operations introduction or as a note.

Something like, "Individual operations in a request are semantically equivalent to singular requests performed serially with a transactional guarantee."

Perhaps that could be left to the recommendations page of the website too :)

If order can be elided by the server because of some knowledge it has, then I think it is free to do so. However, there should be no "evidence" in the response, because that breaks the contract with the client. If this allowance requires some tweaking of the language used in this proposal, I'm open to it.

Does this suggest that the imperative ought to be SHOULD over MUST? I'm actually unclear on this aspect of specification-writing. As long as there's no "evidence," as you said, does the imperative matter?

Thanks for taking the time to address concerns @dgeb ❤️

Everything possible with singular requests can be completed with operations, and vice versa.

This is why I favor mirroring sideloading payload - why not mimic what we do in singular requests (RESTfully pair a resource with a verb) all the way down, instead of introducing a new payload?

• There is no additional knowledge required by clients of how a server persists resources
• If order can be elided by the server because of some knowledge it has, then I think it is free to do so.

I'm not sure how these statements are reconciled with the statement in the PR "A server MUST perform operations in the order they appear in the operations array". Doesn't this mean the client now needs to know the correct order to persist resources? Or is this just a matter of updating the PR language?

If order is not important, then I would strongly recommend making multiple singular requests in parallel. By using operations, the client is saying that order is important.

This is often not possible, for instance when things need to run in the same transaction but the order doesn't matter - most nested web forms with validation errors would fall into this category. Time-consuming ETLs is another use case I have personally run into.

In fact, none of the examples given in the PR seem order-dependent (at least not for the >90% use case):

• Creating an invoice together with one or more invoice items.

Most servers would create the invoice, then create the items, then roll back if anything is invalid.

• For example, removing two members of an organization at the same time two other members are added.

Note sure why the order would matter for the default use case.

• Deleting an author and some of the articles associated with that
  author.

Most servers would delete the author, then the articles, within a transaction.

Considering this PR makes a great case for supporting common use cases as part of the base spec, and ordering of operations is not a common use case, wouldn't it make more sense to have "sideposting" in the base spec and operations as an extension? Assuming we don't want to go with the option of simply adding an order key.

Sorry, I rushed my comment as I was going out the door and realized I should retract this statement:

If order is not important, then I would strongly recommend making multiple singular requests in parallel. By using operations, the client is saying that order is important.

This would be better stated:

If order and atomicity are not important, then I would strongly recommend making multiple singular requests in parallel. By using operations, the client is saying that order and / or atomicity are important.

@richmolj Thanks for checking me on this point. I'll respond to your other points soon.

This is why I favor mirroring sideloading payload - why not mimic what we do in singular requests (RESTfully pair a resource with a verb) all the way down, instead of introducing a new payload?

In the side-posting issue you have proposed modifications to the document structure to support layering capabilities on top of side loading. For instance, you proposed adding op or method to individual resources so that you can, for example, delete some resources as you add others. You have to be extremely careful with HTTP compliance here - only PATCH allows for such flexibility for media types.

I'm not sure how these statements are reconciled with the statement in the PR "A server MUST perform operations in the order they appear in the operations array". Doesn't this mean the client now needs to know the correct order to persist resources? Or is this just a matter of updating the PR language?

I said above that I am open to tweaking the PR language. The important thing (to me) is that the client not be able to distinguish that a server has chosen to parallelize processing some operations.

In the current spec, clients already need to understand order dependencies to some extent because servers are free to impose any order dependencies they want. Some servers will require that invoices be created before invoice items, for instance. The specification of allowed ordering of operations seems orthogonal to this debate and is probably best solved with a capabilities specification (something we should definitely tackle).

I should also point out that sideposting can have related concerns. A server may allow sideposted invoice items together with an invoice as primary data, but not the other way around. Ultimately, as a specification, we have to allow servers to control of what's allowed at each endpoint. Again, I think this can be best communicated with a capabilities specification.

@dgeb I actually think we agree on most things.

The specification of allowed ordering of operations seems orthogonal to this debate and is probably best solved with a capabilities specification (something we should definitely tackle).

I agree! Let's put ordering to the side for now ✅

The important thing (to me) is that the client not be able to distinguish that a server has chosen to parallelize processing some operations.

Agree the client should be agnostic in this regard ✅

only PATCH allows for such flexibility for media types.

While I am not sure I agree with this interpretation, I'm happy to agree on PATCH being the only verb used for operations/sideposting ✅

A server may allow sideposted invoice items together with an invoice as primary data, but not the other way around. Ultimately, as a specification, we have to allow servers to control of what's allowed at each endpoint.

Agree on this was well - just as an invoice can sideload invoice items, but not the other way around, there should be no such expectation for a sidepost. ✅

lid

Yep, sounds like we need this no matter what ✅

Given all the above, it looks like we have two choices to solve this problem:

• Mirror the existing sideload payload, adding a single key ("sideposting")
• Introduce a completely new payload ("operations")

Since we seem to agree on so much, I'm wondering how we are reaching different conclusions on this final choice. What am I missing?

@richmolj Just want to say that I love your emphasis on consensus <3

I'll get back to you soon with some examples that I hope will clarify any differences that remain.

@richmolj Ok, below is my attempt to distinguish between sideposting, as I believe you're proposing it, and operations as proposed in this PR. Please feel free to correct any misunderstandings about your proposal.

I'm pretty sure it's an oversimplification to say that sideposting just involves "Mirror the existing sideload payload, adding a single key". I think what you're proposing is allowing an op member to be associated with any resource and resource identifier in a compound document. I'm not sure "sideposting" is an accurate name for your counterproposal - maybe "graph mutations" would be more accurate? Anyway, I'll call it sideposting for the sake of discussion.

I believe that sideposting would involve the following changes to the spec at a minimum:

• If only PATCH is used for sideposting, then every resource and resource identifier in the compound document request would need to include the op member to understand its role in the request. Or there would need to be some concept of a "default" op for the request.

• We would need to make this breaking for 1.0 servers which do not understand sideposting. This could be handled via version or feature negotiation or somehow choosing a structural distinciton that would make these requests invalid for 1.0 servers. Keep in mind that 1.0 servers are supposed to simply ignore the op member and wouldn't be expecting anything in included in a request.

• New rules would be needed for responses to guarantee that every lid sent in a request can be mapped to an id in the response. Something along the lines of the identities array would be needed as I explored for a bit in Introduce Local IDs to v1.1 #1244 (before removing it). Without this guaranteed mapping, clients risk never being able to map local ids to server ids.

In addition, to fulfill some of the other use cases solved by operations, the spec would need to expand in additional ways:

• In order to allow multiple resources of the same type to be created / updated together, we'd have to expand the definition of primary data to allow it to be an array of resources, and not just a single resource.

And then there are some general implementation complexities:

• Since the graph of resources could be arbitrarily complex (but ultimately limited by the server), a server would need to generate execution plans for each allowed graph based upon the operation associated with each resource or resource identifier. For instance, the server might need to delete some resources before updating others. Cross references between resources would need to be fully accounted for. Basically, the full graph of primary and included resources would need to be traversed before an execution plan could be generated.

• Clients would need to track all mutations based upon relationships. If a mutation could not be related in one way or another, they could not be included in the same request. There could be substantial bookkeeping on the client.

Åfter all this, sideposting still wouldn't be able to handle use cases like the following:

• Create three articles and then fetch the first ten articles sorted by title in a single request.

• Collect mutations which may or may not be related and perform them together transactionally (e.g. while a client is offline).

To contrast, the operations proposal does introduce a new wrapper object, the operation object, but the rules for processing each operation, and the capabilities allowed for each, are identical to the rules for singular requests. The only additional requirements are accounting for local IDs and transactionality (but these requirements are also present in the sideposting proposal).

In sum, I consider the operations proposal to be the smallest delta from the current spec, while also providing the greatest range of new functionality.

Thanks for taking the time to give a detailed breakdown @dgeb, I really appreciate it ❤️ Though I'll still make the case for sideposting below, I think this will lead to some constructive edits and a healthier PR at a minimum.

Re: Sideposting

We would need to make this breaking for 1.0 servers

As the changes are purely additive (and the server should ignore the additions, as you say), how is this breaking? Seems in-line with a minor version release to me.

every lid sent in a request can be mapped to an id in the response

Because this is a mirrored payload, the lid would exist in the response at the same level (in the relationship resource identifier and next to the server-minted id). This is how clients map lid to server-generated ids.

allow multiple resources of the same type to be created / updated together

I think this is an independent issue. "sideposting" would be compatible with an array of data, for instance, which is something that could be merged independently of this discussion. Relatedly: see the "Operations: B" note below, where I note this might be a better fit for an extension than the base spec.

As a minor note (don't want to get held up on this point), sideposting would support this even without that independent PR like so:

PATCH /bulk

{
  data: {
    type: "bulk_updates"
    relationships: {
      articles: [
        { lid: "a", op: "create", type: "articles" },
        { lid: "b", op: "create", type: "articles" }
    }
  }
  included: [...article data...]
}

server would need to generate execution plans for each allowed graph based upon the operation associated with each resource or resource identifier

I believe that this is a feature, not a bug. Baking this logic into the client means coupling the client to server implementation. Taking the example "For instance, the server might need to delete some resources before updating others." - what if this was true in January, but the reverse is true in December after I have swapped my datastore? I should be able to keep the same API without requiring all clients to be updated.

I think this comes back to the "A server MUST perform operations in the order they appear in the operations array" language in the PR that @gabesullice and I commented on above, and is my chief concern with operations. If we removed the "Must" here it seems both solutions would have that same server-side challenge, if we keep the "Must" it seems like it would lead to client/server coupling.

Clients would need to track all mutations based upon relationships

I'm unsure what you mean by this, could you give an example? How is this different from sideloading, which is also based on available relationships?

Again, I think this might be an independent issue regarding single-request writes of unrelated data.

Re: Operations

Create three articles and then fetch the first ten articles sorted by title in a single request.

Collect mutations which may or may not be related and perform them together transactionally (e.g. while a client is offline).

On the first, I agree, sideposting would not support this. On the second, I somewhat agree but as noted above I think this is a separate PR possibly unrelated to operations or sideposting.

In any case, I'm guessing you separated these out into a separate section because they are less common scenarios. If these two points are what separates the abilities of operations from the abilities of sideposting, it would make sense to

A) Update the PR to explicitly call this out as a requirement we are trying to solve
B) Move operations to an extension and sideposting into the base spec

The reason I arrive at B: per the PR description, this was originally an extension or separate media type, but the scenarios were too common to ignore. I actually strongly agree with this when we are talking about updating primary data alongside relationships in a single request (as "sideposting" solves)...but disagree when we are talking about blending reads and writes into a single request (which "operations" solves, amongst other important but less-common use cases).

As an aside: the benefit of "sideposting" is the intuitive mirroring of "sideloading", which we generally expect from RESTful APIs that are built to transfer the same state representation back and forth to the server. There is no question that "operations" is a more powerful paradigm - but in my opinion it diverges enough from expectations that it makes more sense as an official extension.

The lack of bulk operations (mostly bulk updates) severely hinders adaptability of JSON API in conversations I've had about standardizing API wire formats. I'm excited to see progress made on this front and I very much agree that something like this should go in the base spec.

I like that this doesn't really add much in terms of functionality. It appears to be a small superset of the existing functionality, right? It's essentially a tool for optimizing for less round trips to the server. It has me wondering: if operations can essentially do anything the rest of the spec can do and one of the huge benefits to a standard wire format is that standard tools on each end can understand it, what need do we have for the spec's top-level document structure definition keys (data, included) or the use of HTTP VERBS at all?

For my use of JSON API, it's an explicit goal that I rarely had to read the actual request/response because the tooling on each side does such a good job abstracting that away. That tooling could be simplified if it always used operations, right?

@richmolj I'm not sure whether to keep this conversation going here. Maybe a separate issue would be better to allow operations to be refined here? But anyway, I appreciate your responses and will follow up as much as time allows:

As the changes are purely additive (and the server should ignore the additions, as you say), how is this breaking?

We would want 1.0 servers to treat sideposted requests as breaking because they would attempt to understand the payload with a PATCH method as a resource update request, and they would ignore the op code in the resource. In other words, 1.0 servers would misinterpret sideposting because they have no concept of this new feature. We don't want clients to get a false "success".

Because this is a mirrored payload

Are include or sort params allowed to customize the response? Or is it a hard rule that the response directly mirrors the request?

I think this comes back to the "A server MUST perform operations in the order they appear in the operations array" language in the PR that @gabesullice and I commented on above, and is my chief concern with operations. If we removed the "Must" here it seems both solutions would have that same server-side challenge, if we keep the "Must" it seems like it would lead to client/server coupling.

I've responded above that I'm fine to tweak the MUST language: "The important thing (to me) is that the client not be able to distinguish that a server has chosen to parallelize processing some operations." Note that this would allow servers to create an optimized execution plan, but not require it.

I'm unsure what you mean by this, could you give an example?

Clients would need to track dirtying of record graphs instead of tracking a queue of changes (i.e. operations).

I'm guessing you separated these out into a separate section because they are less common scenarios.

The other sections were related to adjustments that would be required to fulfill sideposting as you're describing it. I separated these items out because I have not heard any proposals related to sideposting that attempt to address these scenarios.

@EndangeredMassa Thanks very much for your feedback!

I like that this doesn't really add much in terms of functionality. It appears to be a small superset of the existing functionality, right?

That's correct.

That tooling could be simplified if it always used operations, right?

Yes, tooling could use operations exclusively without losing any functionality from the current spec.

Really sad to not see this in 1.1 :( What's the reason for this? What is open to get this (Operations) finalized? And what is the (rough) timetable for 1.2? This has been open for a pretty long time now, so it would be really great to finally have these capabilities in JSON:API.

Really sad to not see this in 1.1 :( What's the reason for this? What is open to get this (Operations) finalized? And what is the (rough) timetable for 1.2? This has been open for a pretty long time now, so it would be really great to finally have these capabilities in JSON:API.

@mydea from my perspective, operations fills one of the most pressing gaps in the spec, so I understand your feelings. Simply put, we decided to ship 1.1 with a smaller set of features rather than delay it further, since profiles will bring a lot of value on their own. We will resume discussing 1.2 this week with the understanding that operations are now the highest priority. I think it's possible that we could ship a 1.2 RC within a few months of 1.1 final.

I've skimmed through this a bit, and wonder how would this look with something like "belongs to" relation? If the client is creating a Place and in the same window/form he can create a City (through a modal, like a + button near city select etc.), so Place belongs to City and both should be persisted on save. In this scenario City must be created before Place, but if this is handled on the client side, then the client needs to know about what kind of relationship this is (and all the others) and make implementations/execution plan based on it. From developers standpoint, this would be a "disgusting" API to work with? What would be the go to in this scenario? From my point of view, the server should be the one to build the execution plan, otherwise this leaves API's frustrating to use if anything.

Hi @dgeb,

Could you provide us with an update regarding this pull request and/or version 1.2?

Kind regards

Oh god! This specification would solve my problems and make me sleep very well!

But I guess that we are too far from that, because this is just for v1.2 and the v1.1 that was planned for January 31, 2019 is not even finished 😕

Is there any timeline on this? Frankly, this has been long overdue and is pending without any clear timeline for 1,5 years. We are using JSON:API heavily, but this issue is massively holding us back, which is driving us to consider other API designs. Which is a shame, since we really like the general idea of JSON:API.

I would just like to have a proper understanding of where this is going. If we cannot, realistically, consider this to be finalized e.g. in the next year, then we'll be forced to look for other options to solve this (IMHO probably pretty common) issue of transactionality.

Any update as to when this will be implemented?

Going to just poke at this again. Any comments from the maintainers? Is this even being discussed?

I'm closing this PR in favor of #1437, which introduces operations as a new v1.1 "extension" (see #1435). The atomic operations extension is technically very close to this PR, with the main exception being that it covers mutations only and not get operations. It will likely be the only official extension launched with 1.1, since local IDs have already been merged into the 1.1 base spec.

Thanks to everyone for the discussions and focus on this over the years. We're very close to being able to use operations "officially" now.

Sorry, to comment on a closed PR.
Does that mean v1.1 have the ability to

Creating multiple resources of the same or different types together and then
creating relationships between them. For example, creating an invoice
together with one or more invoice items.

On documentation, I think it didn't point to that direction.
https://jsonapi.org/format/#crud-creating