I merged one or two of the other PRs, can you rebase this real quick 😭

Also, @dgeb maybe put a link to https://github.com/dgeb/json-api/blob/v1rc1/format/index.md at the top so we can see the rendered version more easily?

Disclaimer: I came here specifically because I'm invested in the error API.

"id" - A unique identifier for this particular occurrence of the problem.
"href" - A URI that MAY yield further details about this particular occurrence of the problem.
"status" - The HTTP status code applicable to this problem, expressed as a string value.
"code" - An application-specific error code, expressed as a string value.

I am super happy with the work here except for the above. I seriously believe these are 4 ways of saying the same thing. Though I do admit href's similarity is arguable, but a uri is a string to an error documentation and by definition of being a uri it's an identifier.

id and code are literally the same thing: Both application specific (though id isn't documented as such for some reason), both probably going to be strings in the real world, both specific to the error at hand.

And I've just never liked repeating HTTP Status Codes in the body of my message. It's in the header already.

I can certainly see that. In my mind, id is what your code checks, and code is what the programmer shows, to help them understand.

if id != 9001
   throw code

Does that make sense? Then again, you're right, the client could

if id != 9001
    throw http_client.get(href)

But that seems excessive.

And I've just never liked repeating HTTP Status Codes in the body of my message. It's in the header already.

Right, you'd have something more specific. '409 conflict' comes with a body that says 'you forgot the required value foo'.

Does that make sense? I can certainly see it your way, just laying out some of the why.

Oh, also cc @mjallday and crew (it won't let me @balanced/engineers here for some reason), since Balanced is one of the bigger deployments of JSON API. AFAIK, they should still be compatible, just using some MAYs rather than some SHOULDs.

So it's HTTP status codes for individual leaves (fields in Rails case)? Bleergh, like most status codes don't apply for that sort of thing. Seems like shoe horning, but alright, maybe that's a thing that needs to happen.

That said, throwing an MAY on the code and href is a priority, or id and href. See? I firmly believe id and code are super interchangeable OR code gets subverted in your example (developer specific notes) to be the way the server sends error messages ("Field needs to have 3 to 5 characters").

Eh, maybe it's no big deal, but I err on the side of cut these days.

I'm using the HTTP codes as an example, not that they should be HTTP status codes. They're application-specific codes, as I think the spec mentions.

Eh, maybe it's no big deal, but I err on the side of cut these days.

+1, I wouldn't mind MAYing them.

Yeah, okay, then I revert to my original status, code, id, (maybe) href are all ways of saying "Hey, this is the type of problem". MAYing all but one would be ideal IMHO. Thanks for listening.

I'm a little concerned with the change that a singular resource still would have a plural "key". This seems inconsistent with the fact that one to one relationships have a singular key.

It also seems inconsistent that an error resource should be an array always where as normal resources should be an object.

I'm a little concerned with the change that a singular resource still would have a plural "key".

This is basically for backwards compatibility with previous versions. You SHOULD use data as a key now instead.

It also seems inconsistent that an error resource should be an array always where as normal resources should be an object.

Hmmmmm.

This is basically for backwards compatibility with previous versions.

Backwards compatibility as part of a BREAKING CHANGE of having the singular resource be an object? Can you please expand on that future as to what compatibility exactly this maintains?

You SHOULD use data as a key now instead.

For all resources? My reading made it sound like only variable type resources should have the "data" as a key. Singular type resources SHOULD have either the type as the key OR data as the key (with type based key listed as the first option)

@krainboltgreene Thanks for your feedback re: errors.

All of the members of an error object have a unique purpose and all are optional.

id and code differ in that id represents this particular occurrence of the problem
(if your app wants to track each and every error that occurs), while code represents
a particular error type. It's like the difference between a class and an instance.

href and id could in theory be identical, but it's not required. href should be used for lookups.

status is available primarily for bulk operations, as described in the "Other responses" section:

When a server encounters multiple problems from a single request, the most
generally applicable HTTP error code should be specified in the response. For
instance, 400 Bad Request might be appropriate for multiple 4xx errors or 500 Internal Server Error might be appropriate for multiple 5xx errors.

While a general 400 code might be returned when more than one operation fails, more specific status codes can be included in each individual error object.

@steveklabnik - @wycats and I backtracked on SHOULD for using data in general. The only time it's an absolute requirement is for polymorphic (i.e. multi-type) collections, in which case each resource SHOULD include its type.

I'm a little concerned with the change that a singular resource still would have a plural "key". This seems inconsistent with the fact that one to one relationships have a singular key.

@kjg oh, the heartburn that this minor issue has given me over the past few months.... We tried every manner of approach to resolving the singular / plural inflection issue and every solution had a problem with it (which I should write up thoroughly). We came to the conclusion that this is not about singulars and plurals as it is a string that represents a "resource type". For some APIs this might be singular; for some plural. Because most URLs reference plural collections, all the examples use the plural form.

One of the solutions was to simply use "data" everywhere, which has the distinct drawback in that it no longer identifies the type of the primary resource. For this reason, any use of "data" MUST be coupled with a type identifier in the resource object. The goal here is to ensure that a document can be interpreted on its own, without the context of a URL, and therefore needs to identify the type of each resource contained within.

For this reason, any use of "data" MUST be coupled with a type identifier in the resource object. The goal here is to ensure that a document can be interpreted on its own, without the context of a URL, and therefore needs to identify the type of each resource contained within.

Independent of the use of the "data" key, I’d still push that JSON API should highly recommend people include a "type" key in each resource. In my experience, it’s very convenienent to rely on "id", "type", and "href" residing within a resource’s object body.

@dgeb Alright, those explanations worked for me, though I'm not sure I'm cool about status just yet. Time will tell I suppose.

@andrewsardone Wouldn't this be rel in web speak?

@krainboltgreene: just like in Variable Type Resource Representations, I think plain ‘ol resource objects should include a "type" key:

{
  "posts": {
    "id": "1",
    "title": "Rails is Omakase",
    "type": "posts"
  }
}

(I would also always include the "href" key)

I'm philosophically opposed to forcing people to include a type unless it's required to resolve an ambiguity. HTML doesn't require you to type its responses, nor does regular old JSON.

If we have to, we have to, but I think it's best if it's optional.

@steveklabnik I agree.

To be clear: in this PR, type is only required in individual resource objects when it can't be determined from elsewhere in the document (i.e. collection keys, top-level links object, etc.).

Of course, individual implementations may choose to include type in every resource object for convenience, even if it's redundant.

Eh, really disappointed to see reversion back to polymorphism. Is there rationale besides "It matches requests"?

Strongly-typed APIs are really nice to work with.

As for the errors, I note that we've stripped the ability to say "this part of your request is bad", leaving us just with "your request is bad and you should feel bad." Kind of disappointed about that, too. Not sure why we wouldn't point to a specific field or parameter using JSON pointers to say "here's what's wrong".

Otherwise, has anyone tried actually building--for example--a simple registration form with these error constraints? How do you display to the user that the "email address" field is not a valid email format? How do you display that the username has unacceptable characters? Are we really suggesting that codes should be unique to errors and fields?

I've tuned out a bit, so I'm probably missing a lot, but we seem to have some compromises here that just... don't make sense. The errors object specifically looks like design-by-committee instead of an actual purpose-driven design.

@paddyforan this is exactly why this PR is open. Let's work through that stuff! The previous stab at errors was too simple, and instead of not causing bikeshedding, caused bikeshedding. Nothing here is sacred!

After a bit more consideration, my reservations on errors is that it tries to do a great many things and does none of them well.

The data being returned as "an error", as far as I can tell, is meant to:

• tell the application developer what went precisely went wrong at runtime (the error code). Unfortunately, this information isn't actionable at all unless a preposterous number of codes are created. How many different types of invalid user input does your API have? Because under this scheme, each of them needs its own unique code. Yikes.
• tell the user how to refer to their problem when contacting support (the id). Unfortunately, including this alongside the code without some very well-done explaining is going to confuse people... as we've seen above.
• tell the user and/or the client library developer where to seek more understanding about the error (the href). Unfortunately, without a clear indication of what that href links to, it's essentially useless. Does accessing that href give me more machine-readable information to present to the user? Does it give me a direct link to the API reference I violated so I can fix my mistake? Who is it meant for, and in what context?
• provide an error string, with the same problem: without knowing the audience of these, client libraries will have no idea whether to pass them on to the user or log it.

Has anyone actually tried building an application that consumes an API that would meet these specifications? It seems very wishy-washy to me and hard to get right at this point, but we're going to pin a 1.0 on it?

If I had to make a suggestion for the errors, I'd say let's pick one or at most two purposes for the error returned and make sure it does a great job of them. Which means not everyone will get what they want, I imagine, but making errors good at something is probably better than making them good at nothing.

@paddyforan - thanks for the feedback!

Eh, really disappointed to see reversion back to polymorphism. Is there rationale besides "It matches requests"?

The default approach in this spec is to use strongly typed resources. However, I see polymorphic collections and relationships as a reality for many APIs, so it's important for the spec to address them.

A common example is an activity stream that comprises multiple types of activities.

As for the errors, I note that we've stripped the ability to say "this part of your request is bad", leaving us just with "your request is bad and you should feel bad."

Actually, each error resource can provide specificity to the developer and the end user regarding the error:

The developer can use the links member to discern exactly which resources in the request are related to this error (e.g. "links": {"posts": "123"})

The path member can be used to specify the relevant attribute for this error (e.g. "path": "/title")

The code can be used by the developer when responding to this error (e.g. `"code": "1234" - client code might know that this means that an attribute can't be blank). It wasn't my intention for this to be presented to the user.

The title can be displayed to the user (e.g. "Title can't be blank") as can detail (useful for more complex errors).

The id could be referenced by the user when contacting support or by the developer when logging problems.

I really should expand this section with examples, which I think could make these choices more clear. I'll try to come up with some soon - I think they'd be worthwhile in the spec.

Has anyone actually tried building an application that consumes an API that would meet these specifications? It seems very wishy-washy to me and hard to get right at this point, but we're going to pin a 1.0 on it?

I share your reservations about hardening this spec immediately after so many changes. My preference would be to treat this as a release candidate. If this looks like it's 90% of the way there, let's merge it, kick the tires over a set period, and incorporate feedback. Of course, any feedback before a merge is welcome too :)

@paddyforan Your point is well taken regarding the tension between these design goals. I've added a clarification.

@ahacking I've also added a clarification regarding resource URLs as well. The same rules apply to determining URLs and PATCH paths - they are complementary when forming JSON Pointers to identify a resource.

Yes those discussions are exactly what I was getting at. You need patch compatible structures.

This seems to require client assigned ids such as v4 uuids since the client can't invent a server assigned id unless a special client id value space is reserved which hopefully does not clash with existing ids. If anyone has ids based on say name or title this can never be workable without an escaping/encoding mechanism eg prefix client ids with '_' and if any id client or server has leading underscores then use double underscore for all leading underscores. This always means an odd number of leading underscores for client ids.

The use of an api which allows client assigned ids has different usage than one that uses server assigned ids. With v4 uuids its the cleanest possible solution, with cids you have an additional mapping for whatever the server assigns and it can make web clients that have browser urls for resources behave in unexpected ways as cid's are typically ephemeral and unlikley to be persisted.

So where does JSON API draw the line?

@ahacking please see the sections on IDs and client-generated IDs.

@dgeb yes that confirms the issue for me.

If collections are represented in a patch compatible fashion then both 'clientid' and 'id' share the same value space as the property names on the object representing the collection.

There needs to be a way to distinguish and separate clientids from server ids so clashes never occur when using patch compatible structures.

@ahacking This seems to be an pretty narrow problem that should only come up when performing a bulk patch operation in which a resource is both created and then updated or deleted in the same request.

The creation / deletion scenario seems really rare and slightly nuts:

PATCH /
Content-Type: application/json-patch+json

[
  { "op": "add", "path": "/photos/-", "value": {"title": "beach", "clientid": "a"} },
  // ...
  { "op": "remove", "path": "/photos/a"} }
]

The creation / update scenario seems more likely as part of a bulk sequential update:

PATCH /
Content-Type: application/json-patch+json

[
  { "op": "add", "path": "/photos/-", "value": {"title": "beach", "clientid": "a"} },
  // ...
  { "op": "replace", "path": "/photos/a/title", "value": "I need a vacation"} }
]

The far more common scenario is that multiple resources are created and linked in the same request:

PATCH /
Content-Type: application/json-patch+json

[
  { "op": "add", "path": "/photos/-", "value": {"title": "beach", "clientid": "a"} },
  { "op": "add", "path": "/notes/-", "value": {"clientid": "b", "text": "I need a vacation", "links": {"photos": {"clientid": "a"}} }}
]

In the last scenario, "clientid" can still be used to establish links without needing to use it in a URL / path.

So, what do we do about the first two scenarios?

• A) Not allow them in the first place by not allowing clientids to be used in URLs / paths. These ops would only be allowed with UUIDs, identified as "id".
• B) Defer to clientid when identifying any id, since a map must already be maintained by servers that support clientids.
• C) Use a prefix, such as 'clientid:a' when referencing clientids in URLs / paths (I know, this is done already for heterogenous collections).

I do think that this is a pretty narrow problem best solved in its own issue and not a showstopper.

This thread is growing quite long. My preference would be to merge this sooner than later, treat it as an RC1, and commence discussions in more focused issues. I have a very busy week coming up, starting now, and won't be able to be as active here as I have been.

I'm fine putting this on ice for a bit as well if everyone wants to give it more time / consideration. I still want to consider some of the finer points myself, like the clientid issue above, which I don't see as a blocker.

@steveklabnik @wycats - thoughts?

@gdeb I am thinking the simplest use case of adding a resource with a related resource using PATCH with patch compatible structures (ie no collection arrays, just objects with ids as keys).

[
  { "op": "add", "path": "/photos/_a", "value": {"title": "beach", "clientid": "a"} },
  { "op": "add", "path": "/notes/_b", "value": {"clientid": "b", "text": "I need a vacation", "links": {"photos": ["/photos_/a"]}}}
]

Note the use of JSON Pointers with a patch compatible collection approach using clientid values as keys, instead of a collection array. Also the use of links with a JSON pointer.

When you use the 'id' or 'clientid' value as a key in the JSON Pointer then you have to worry about clashes between real ids and client ids and need a mechanism to separate and distinguish them. I suggested an example prefixing (somewhat private) client ids with underscore to distinguish from real ids in JSON Pointers. However you need to handle cases where ids actually have a leading underscore, hence suggesting use of double underscores on leading underscores as an escape mechanism. By this logic a client id will always have an odd number of leading underscores. I don't particularly care what the escaping mechanism is, just that you need one if you want to use JSON Pointers with combinations of real ids and clientids.

I don't like any of this really, but JSON Pointer and JSON Patch are what they are and using array indexes to reference collection items is unworkable outside of a single resource and is still a problem if the resource has changed.

PATCH needs to work against a reference model where collections are objects, which means collections should be objects and not arrays, and links should be JSON Pointers too.

The other option is borrow what you can where sensible from those RFC's but do what actually works best and report findings into the RFC process. From where I stand JSON Patch and JSON Pointer have some severe assumptions and limitations and we are trying to work around it but is not exactly clean.

I am witnessing all the XML things being repeated in JSON... here we go again...

@ahacking You raise good points.

The other option is borrow what you can where sensible from those RFC's but do what actually works best and report findings into the RFC process. From where I stand JSON Patch and JSON Pointer have some severe assumptions and limitations and we are trying to work around it but is not exactly clean.

Yes, we are bending JSON Patch and Pointer to our purposes but aren't completely compliant if we are adding to /posts/- (an array path) instead of /posts/_a (an object path) and then accessing that same resource at /posts/_a. We should be clear-eyed about whether we want to break compliance like this.

I don't particularly care what the escaping mechanism is, just that you need one if you want to use JSON Pointers with combinations of real ids and clientids.

Yep - the three options above

@dgeb Another thing that I wasn't clear on was at what point does the clientid get replaced with the servers id?

If the response returns objects, do those objects use the clientid or the server assigned id as the key when a JSON Pointer compliant structure is used? If the collection key on a response always represents the objects id does the id also need to be carried in the actual object as well? (it just seems like extra overhead and more chance to get an inconsistency). Obviously clientid should always be included in the object.

My thinking is that if I do a POST/PUT/PATCH and the created/updated objects are returned in the response and then I later do a GET, I would like the response to be consistent aside from carrying a clientid if one was given in the POST/PUT/PATCH request. Said another way, always use the real id as the collection key when the object has one. It begs the question what to do when objects don't have ids, at that point we need something to use as a collection key, perhaps an index/position number?

@ahacking Any time clientid is passed to a server, the server should return it in the resource object. That's the only place it should appear in the response.

It begs the question what to do when objects don't have ids, at that point we need something to use as a collection key, perhaps an index/position number?

Objects that don't have ids should never be referenced as members of a collection.

@ahacking @dgeb

Hmm, just to let you know, gdeb != dgeb... Also, hello dgeb, nice nickname!

hey @gdeb - sorry that you got dragged into this :)

The more I look at this the more I am tempted to say EVERY object has a v4 UUID and go and play somewhere else if you want to make life hard. But I realise that such a hard nosed approach requires the database to use v4 UUIDs as their primary keys or to carry (v4) UUIDs. You can't easily add JSON-API support to existing datasets but if JSON-API is setting a good example maybe that is a sensible line in the sand.

If I was to profile JSON-API without a doubt I would mandate (v4) UUIDs everywhere because of the clear advantages at all levels of the stack, they are invariant, clients can build complex object graphs, servers can easily shard for scalability.

I think JSON-API should either say all resources have a (v4) UUIDs, there are no client ids, OR we bear the cost of maintaining two profiles, JSON-API and JSON-API-LIFE-IS-HARD. I think it is important if you want to support the life-is-hard approach then structure the spec as two separate profiles to keep the easy way clean and the all the ifs buts and maybes in the life-is-hard spec.

@gdeb sorry... was messaging on my phone without the nice auto-complete.

@ahacking I agree that v4 UUIDs are simpler conceptually and provide many advantages. I was trying to provide a bridge for implementations that don't support UUIDs, but this approach does have its own costs for implementation.

If this non-UUID clientid solution is seen by you and others as too half-baked and/or complicated for inclusion, I'm fine with removing it from the PR and we could discuss it separately.

As far as the earlier discussion re: PATCH / Pointer, there really is no good alternative to allowing add ops to the end-of-array path (e.g. /posts/-) if we want to support scenarios in which the client does not generate ids.

@dgeb. Agreed. I will always use uuids but I really would like a simple one way of doing things. Nail the plural singular thing (I prefer plural always as it matches endpoint names) I always want a collection object keyed by id as the definitive reference model given we want to use JSON pointer. I want this even for singular resources. Uniform structure always regardless of cardinality.

I am happy to have links in a nested object so that generic processing logic can wire things up.

I think there is a need for meta info in objects however and that is how I have carried errors to date in my APIs but I'll go along with problems and JSON pointers. I do want the ability to have custom attributes in my errors/problems which could be useful depending on the type of error (eg could be suggestions for alternative names or other hints for validation errors).

I am really wanting simplicity, one way for everything and minimum repetition hence not wanting id in the object but on the collection key since it has to be put there for JSON pointer to work generally. Then we have a uniform structure, just resources housed in collection objects which are housed in a root object and JSON pointers to link up the object graph.

As for polymorphic concerns I think let users define whatever named collections they like and don't worry about prescribing types. Its only when you describe a generic data collection in the spec that it became a concern. I would stay silent on it. If people want to use a brown bag let them and let them decide how to distinguish types. The id will need to be unique across types because all types will share the same key space when they are all in the same object and that is again another win for (v4) uuids and a whole lot of stuff JSON API can stay out of.

Thanks for everyone's feedback. I'm going to explore a simplification of this PR in a new branch, which may not be complete for another week or more. I do think that there's some fat to be trimmed from the spec and some ideas that could be more fully baked. I'm primarily concerned with polymorphism and client ids.

Please feel free to continue to comment here or on other issues. As I mentioned above, I've got a busy week ahead and may not be as responsive as I've been.

@dgeb Cheer's, I look forward to seeing the updated spec

I need an example of what an error response looks like. I wasn't able to reason with the given description.

Please review #237, which represents the simplification of this PR that I promised. The differences are contained in dgeb@8f831e9

Closing in favor of #237