JSON Duplicate Keys

classic Classic list List threaded Threaded
46 messages Options
123
Reply | Threaded
Open this post in threaded view
|

JSON Duplicate Keys

Douglas Crockford
The JSON RFC says

     The names within an object SHOULD be unique.

Sadly, there are people in the JavaScript community who interpreted SHOULD to mean DON'T HAVE TO. In a perfect world, we would change the SHOULD to MUST, but we can't. Interoperability and security concerns demand that we specify what happens when keys are duplicated. So we may end up with something like this:

     The names within an object SHOULD be unique. If a key is duplicated,
     a parser SHOULD reject. If it does not reject, it MUST take only the
     last of the duplicated key pairs.

Does anyone see a problem with this?

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Anne van Kesteren
On Thu, Jun 6, 2013 at 12:29 PM, Douglas Crockford
<[hidden email]> wrote:

> The JSON RFC says
>
>     The names within an object SHOULD be unique.
>
> Sadly, there are people in the JavaScript community who interpreted SHOULD
> to mean DON'T HAVE TO. In a perfect world, we would change the SHOULD to
> MUST, but we can't. Interoperability and security concerns demand that we
> specify what happens when keys are duplicated. So we may end up with
> something like this:
>
>     The names within an object SHOULD be unique. If a key is duplicated,
>     a parser SHOULD reject. If it does not reject, it MUST take only the
>     last of the duplicated key pairs.
>
> Does anyone see a problem with this?

SHOULD reject seems too strong given that we know perfectly well we
cannot do that. MAY seems much more reasonable or maybe MUST either
reject or take the last of the duplicated key pairs (decision up to
the parser API specification).


--
http://annevankesteren.nl/
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Douglas Crockford
On 6/6/2013 4:34 AM, Anne van Kesteren wrote:

> On Thu, Jun 6, 2013 at 12:29 PM, Douglas Crockford
> <[hidden email]> wrote:
>> The JSON RFC says
>>
>>      The names within an object SHOULD be unique.
>>
>> Sadly, there are people in the JavaScript community who interpreted SHOULD
>> to mean DON'T HAVE TO. In a perfect world, we would change the SHOULD to
>> MUST, but we can't. Interoperability and security concerns demand that we
>> specify what happens when keys are duplicated. So we may end up with
>> something like this:
>>
>>      The names within an object SHOULD be unique. If a key is duplicated,
>>      a parser SHOULD reject. If it does not reject, it MUST take only the
>>      last of the duplicated key pairs.
>>
>> Does anyone see a problem with this?
> SHOULD reject seems too strong given that we know perfectly well we
> cannot do that. MAY seems much more reasonable or maybe MUST either
> reject or take the last of the duplicated key pairs (decision up to
> the parser API specification).
The current RFC already says SHOULD. We don't have to weaken that. But
we should be explicit about what a parser should do if it decides to
accept away. The sentence

     If a key is duplicated, a parser SHOULD reject.

is not a change. It is implied by the first statement.  The thing that
is a change is the third statement

     If it does not reject, it MUST take only the last of the duplicated
key pairs.
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

François REMY-2
> The sentence
>
>      If a key is duplicated, a parser SHOULD reject.
>
> is not a change. It is implied by the first statement.

I do not agree. A 'should' authoring requirement is not meant to trigger an
abortion from the execution engine.

For example, the CSS flexbox spec says that an author should not use the CSS
'order' property to convey any semantic meaning, but that doesn't mean the
browser will reject the 'order' declaration if it detects the way it's used
is not semantically valid. A 'should' is nothing more than an advice to the
author, not a requirement per se.

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

gaz Heyes
In reply to this post by Douglas Crockford
On 6 June 2013 12:29, Douglas Crockford <[hidden email]> wrote:
The JSON RFC says

    The names within an object SHOULD be unique.

Sadly, there are people in the JavaScript community who interpreted SHOULD to mean DON'T HAVE TO. In a perfect world, we would change the SHOULD to MUST, but we can't. Interoperability and security concerns demand that we specify what happens when keys are duplicated. So we may end up with something like this:

    The names within an object SHOULD be unique. If a key is duplicated,
    a parser SHOULD reject. If it does not reject, it MUST take only the
    last of the duplicated key pairs.

Does anyone see a problem with this?

 That doesn't make sense "should" should be "must" because if it's taking the last key then you can overwrite someone else's data.

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Douglas Crockford
On 6/6/2013 5:00 AM, gaz Heyes wrote:

> On 6 June 2013 12:29, Douglas Crockford <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     The JSON RFC says
>
>         The names within an object SHOULD be unique.
>
>     Sadly, there are people in the JavaScript community who
>     interpreted SHOULD to mean DON'T HAVE TO. In a perfect world, we
>     would change the SHOULD to MUST, but we can't. Interoperability
>     and security concerns demand that we specify what happens when
>     keys are duplicated. So we may end up with something like this:
>
>         The names within an object SHOULD be unique. If a key is
>     duplicated,
>         a parser SHOULD reject. If it does not reject, it MUST take
>     only the
>         last of the duplicated key pairs.
>
>     Does anyone see a problem with this?
>
>
>  That doesn't make sense "should" should be "must" because if it's
> taking the last key then you can overwrite someone else's data.
You are exactly right. But it is too late to change to MUST unless TC39
chooses to break the programs of the people who are doing what they
shouldn't, which is something TC39 said it will not do.
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

gaz Heyes
On 6 June 2013 13:19, Douglas Crockford <[hidden email]> wrote:
You are exactly right. But it is too late to change to MUST unless TC39 chooses to break the programs of the people who are doing what they shouldn't, which is something TC39 said it will not do.

Meh politics. I have no interest in such matters. Breaking broken programs shouldn't be a concern.

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Jeremy Darling
Copying in from another post I sent.  Instead of amending SHOULD to allow parsers to throw new errors why not have them emit collections when duplication is found?

Duplicate Keys should never have been allowed in the spec in the first place (most all key/value stores don't allow it) or if they were allowed when decomposed to JS objects they should have created arrays of values much like most frameworks do for duplicate keys in the HTTP headers.

Thus, and typing this makes my skin crawl and my head hurt:

{
  "myKey": "Value 1",
  "myKey": 2
}

Becomes, upon parse:

{
  "myKey": ["Value 1", 2]
}

All values have now been accounted for. nothing has been created or lost, and code that didn't take into account multiple keys but did type checking on values will fail or adapt gracefully.

 - Jeremy



On Thu, Jun 6, 2013 at 8:02 AM, gaz Heyes <[hidden email]> wrote:
On 6 June 2013 13:19, Douglas Crockford <[hidden email]> wrote:
You are exactly right. But it is too late to change to MUST unless TC39 chooses to break the programs of the people who are doing what they shouldn't, which is something TC39 said it will not do.

Meh politics. I have no interest in such matters. Breaking broken programs shouldn't be a concern.

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss



_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Rick Waldron



On Thu, Jun 6, 2013 at 9:31 AM, Jeremy Darling <[hidden email]> wrote:
Copying in from another post I sent.  Instead of amending SHOULD to allow parsers to throw new errors why not have them emit collections when duplication is found?

Duplicate Keys should never have been allowed in the spec in the first place (most all key/value stores don't allow it) or if they were allowed when decomposed to JS objects they should have created arrays of values much like most frameworks do for duplicate keys in the HTTP headers.

Thus, and typing this makes my skin crawl and my head hurt:

{
  "myKey": "Value 1",
  "myKey": 2
}

Becomes, upon parse:

{
  "myKey": ["Value 1", 2]
}

All values have now been accounted for. nothing has been created or lost, and code that didn't take into account multiple keys but did type checking on values will fail or adapt gracefully.

This breaks extant data value invariants. Ignoring the original question and addressing only this example, let's say the expected value was a string (your example shows a string and a number)

  var parsed = JSON.parse(data);
  parsed.myKey.toUpperCase();
  // TypeError

This is web breaking.

Rick
 

 - Jeremy



On Thu, Jun 6, 2013 at 8:02 AM, gaz Heyes <[hidden email]> wrote:
On 6 June 2013 13:19, Douglas Crockford <[hidden email]> wrote:
You are exactly right. But it is too late to change to MUST unless TC39 chooses to break the programs of the people who are doing what they shouldn't, which is something TC39 said it will not do.

Meh politics. I have no interest in such matters. Breaking broken programs shouldn't be a concern.

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss



_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss



_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Rick Waldron
In reply to this post by Douglas Crockford



On Thu, Jun 6, 2013 at 7:29 AM, Douglas Crockford <[hidden email]> wrote:
The JSON RFC says

    The names within an object SHOULD be unique.

Sadly, there are people in the JavaScript community who interpreted SHOULD to mean DON'T HAVE TO. In a perfect world, we would change the SHOULD to MUST, but we can't. Interoperability and security concerns demand that we specify what happens when keys are duplicated. So we may end up with something like this:

    The names within an object SHOULD be unique. If a key is duplicated,
    a parser SHOULD reject. If it does not reject, it MUST take only the
    last of the duplicated key pairs.

+1

As far as I can tell, this is the de facto browser behavior.

Rick


 

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Allen Wirfs-Brock
In reply to this post by Anne van Kesteren

On Jun 6, 2013, at 4:34 AM, Anne van Kesteren wrote:

> On Thu, Jun 6, 2013 at 12:29 PM, Douglas Crockford
> <[hidden email]> wrote:
>> The JSON RFC says
>>
>>    The names within an object SHOULD be unique.
>>
>> Sadly, there are people in the JavaScript community who interpreted SHOULD
>> to mean DON'T HAVE TO. In a perfect world, we would change the SHOULD to
>> MUST, but we can't. Interoperability and security concerns demand that we
>> specify what happens when keys are duplicated. So we may end up with
>> something like this:
>>
>>    The names within an object SHOULD be unique. If a key is duplicated,
>>    a parser SHOULD reject. If it does not reject, it MUST take only the
>>    last of the duplicated key pairs.
>>
>> Does anyone see a problem with this?
>
> SHOULD reject seems too strong given that we know perfectly well we
> cannot do that. MAY seems much more reasonable or maybe MUST either
> reject or take the last of the duplicated key pairs (decision up to
> the parser API specification).

Some that should be pointed out is that one reason JSON needs to accept duplicate field names is that some people currently use them to add  comments to JSON datasets:

{
 "//": "This is a comment about my data",
 "//":  "that takes more than one line",
 "field1" : 1,
 "field2" : 2
}

The above is valid according to the existing RFC and is accepted as valid JSON by the existing ES JSON.parse spec. We don't want to invalid existing JSON datasets that use this technique so duplicate keys MUST be accepted.

What I would say, as a replacement for the current text is:

          The names within an object SHOULD be unique.  If a key is duplicated, a parser MUST take  <<use?? interpret??>> only the last of the duplicated key pairs.

I would be willing to loose the first sentence (containing the SHOULD) entirely as it doesn't add any real normative value.

Allen



_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Allen Wirfs-Brock
In reply to this post by François REMY-2

On Jun 6, 2013, at 4:51 AM, François REMY wrote:

>> The sentence
>>
>>     If a key is duplicated, a parser SHOULD reject.
>>
>> is not a change. It is implied by the first statement.
>
> I do not agree. A 'should' authoring requirement is not meant to trigger an abortion from the execution engine.
>
> For example, the CSS flexbox spec says that an author should not use the CSS 'order' property to convey any semantic meaning, but that doesn't mean the browser will reject the 'order' declaration if it detects the way it's used is not semantically valid. A 'should' is nothing more than an advice to the author, not a requirement per se.

+1

Allen
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Allen Wirfs-Brock
In reply to this post by gaz Heyes

On Jun 6, 2013, at 6:02 AM, gaz Heyes wrote:

On 6 June 2013 13:19, Douglas Crockford <[hidden email]> wrote:
You are exactly right. But it is too late to change to MUST unless TC39 chooses to break the programs of the people who are doing what they shouldn't, which is something TC39 said it will not do.

Meh politics. I have no interest in such matters. Breaking broken programs shouldn't be a concern.

There is nothing broken about then.  People have simply been doing what RFC 4627 allows them to do.

Invalidating existing JSON datasets (many of which may not be actively maintained) is a real concern.  Who does it benefit to make them invalid.

Allen



_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

gaz Heyes
In reply to this post by Allen Wirfs-Brock
On 6 June 2013 16:13, Allen Wirfs-Brock <[hidden email]> wrote:
Some that should be pointed out is that one reason JSON needs to accept duplicate field names is that some people currently use them to add  comments to JSON datasets:

{
 "//": "This is a comment about my data",
 "//":  "that takes more than one line",
 "field1" : 1,
 "field2" : 2
}

That is totally stupid. IMO the JSON spec should support comments like // and /**/ and also support single quotes for string values. It's JavaScript for christ sake. It makes no sense to invent and support this crappy syntax.

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Allen Wirfs-Brock
In reply to this post by Rick Waldron

On Jun 6, 2013, at 6:46 AM, Rick Waldron wrote:




On Thu, Jun 6, 2013 at 7:29 AM, Douglas Crockford <[hidden email]> wrote:
The JSON RFC says

    The names within an object SHOULD be unique.

Sadly, there are people in the JavaScript community who interpreted SHOULD to mean DON'T HAVE TO. In a perfect world, we would change the SHOULD to MUST, but we can't. Interoperability and security concerns demand that we specify what happens when keys are duplicated. So we may end up with something like this:

    The names within an object SHOULD be unique. If a key is duplicated,
    a parser SHOULD reject. If it does not reject, it MUST take only the
    last of the duplicated key pairs.

+1

As far as I can tell, this is the de facto browser behavior.

I'm not sure what you mean by this.  The ES5 spec. for JSON.parse requires (ie MUST accept) that duplicate keys are accepted and that the value associated with the last duplicated key wins.  A valid implementation of JSON.parse MUST not reject such JSON strings.

Allen




_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Allen Wirfs-Brock
In reply to this post by gaz Heyes

On Jun 6, 2013, at 8:21 AM, gaz Heyes wrote:

On 6 June 2013 16:13, Allen Wirfs-Brock <[hidden email]> wrote:
Some that should be pointed out is that one reason JSON needs to accept duplicate field names is that some people currently use them to add  comments to JSON datasets:

{
 "//": "This is a comment about my data",
 "//":  "that takes more than one line",
 "field1" : 1,
 "field2" : 2
}

That is totally stupid. IMO the JSON spec should support comments like // and /**/ and also support single quotes for string values. It's JavaScript for christ sake. It makes no sense to invent and support this crappy syntax.

No it's not JavaScript!  It a file format for storing and interchanging data. Preserving existing data is a key requirement.

Allen


_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Anne van Kesteren
On Thu, Jun 6, 2013 at 4:29 PM, Allen Wirfs-Brock <[hidden email]> wrote:
> No it's not JavaScript!  It a file format for storing and interchanging
> data. Preserving existing data is a key requirement.

Nevertheless, enhancing it with sensible comment syntax would be
great. But as you said, there's a different list for that.


--
http://annevankesteren.nl/
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

gaz Heyes
In reply to this post by Allen Wirfs-Brock
On 6 June 2013 16:29, Allen Wirfs-Brock <[hidden email]> wrote:

No it's not JavaScript!  It a file format for storing and interchanging data. Preserving existing data is a key requirement.

It's syntax is based on JavaScript but crippled for no reason other than to follow a outdated specification. It's so frustrating to be at this end of a keyboard to see awful syntax being promoted just to follow the RFC which is clearly flawed. It's so much more logical to change and fix the problems in the specification so that in future code will work correctly and JSON parsers would be much easier and consistent. "Don't break the web" shouldn't ever become "Keep the web broken" code will never evolve if the core is rotten. 


_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Mark S. Miller-2
In reply to this post by Anne van Kesteren
Everyone, please keep in mind that JSON does not and never has claimed to be the best or last data format. No doubt many better ones have been and will be created, some as variants of JSON, some not. If you'd like to create a "fixed JSON" or "enhanced JSON" or whatever, feel free -- whether it treats comments, duplicated keys, or \u2028 differently or whatever. But please don't confuse any of these variants with JSON itself. As a data format, JSON's greatest value is its stability, and the inability for anyone, including us, to version it.


On Thu, Jun 6, 2013 at 8:45 AM, Anne van Kesteren <[hidden email]> wrote:
On Thu, Jun 6, 2013 at 4:29 PM, Allen Wirfs-Brock <[hidden email]> wrote:
> No it's not JavaScript!  It a file format for storing and interchanging
> data. Preserving existing data is a key requirement.

Nevertheless, enhancing it with sensible comment syntax would be
great. But as you said, there's a different list for that.


--
http://annevankesteren.nl/
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss



--
    Cheers,
    --MarkM

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON Duplicate Keys

Benoit Marchant
In reply to this post by gaz Heyes
What's really problematic is that there is no way to specify a version in the format itself so that it could be evolved without breaking existing data set. This has been solved before for serializetion, in Cocoa for example, we need that for JSON.

Benoit

On Jun 6, 2013, at 8:54, gaz Heyes <[hidden email]> wrote:

On 6 June 2013 16:29, Allen Wirfs-Brock <[hidden email]> wrote:

No it's not JavaScript!  It a file format for storing and interchanging data. Preserving existing data is a key requirement.

It's syntax is based on JavaScript but crippled for no reason other than to follow a outdated specification. It's so frustrating to be at this end of a keyboard to see awful syntax being promoted just to follow the RFC which is clearly flawed. It's so much more logical to change and fix the problems in the specification so that in future code will work correctly and JSON parsers would be much easier and consistent. "Don't break the web" shouldn't ever become "Keep the web broken" code will never evolve if the core is rotten. 

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
123