JSON support for BigInt in Chrome/V8

classic Classic list List threaded Threaded
45 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: JSON support for BigInt in Chrome/V8

Ranando King
What I meant by complete is that if the value isn't an ES Number, isn't a boolean, and isn't null, it will have a string representation, or a binary representation that can be encoded as a string using base64. By using a dataUri for those cases, we can create a consistent library to handle those issues automatically, and without guesswork (no magic). If such a utility API became a standard in ES, other languages would be far more likely to adopt the standardized notation that would be provided.

On Sat, Jul 28, 2018 at 12:03 PM Michael Theriot <[hidden email]> wrote:
I think we should move away from guessing and automagically parsing our JSON. The fact is a JSON number != JS number. Perhaps in userland (or a proposal...) JSON.parse could return with a JSONNumber object which the developer decides to convert to BigInt or Number.

On Saturday, July 28, 2018, J Decker <[hidden email]> wrote:


On Sat, Jul 28, 2018 at 6:57 AM Michael Theriot <[hidden email]> wrote:
In a language with arbitrary integer precision, Python 3 for example, the way to parse a "BigInt" would just be a plain, human readable number without quotes. The way to serialize it is the same. Any other kind of representation is out of spec, a workaround, and belongs in userland.

The problem with this, 'guessing' whether is't a Number() or a BigInt() is that numbers and bigint's don't interop.

{a:123, b:123n}
" { "a":123, "b":123 }"  any expressions that expect B to be a BigInt() will fail, becasue it will be in an expression of other bigints.

bigInts aren't just a better Number type, but, rather require other bigints for their expressions.

 

I think BigInt should serialize the same, not as strings or anything that is not a number. JSON.parse being unable to parse back into BigInt is a separate issue. It is solvable by using better parsing methods, not the convenient built-in one which has other issues. E.g. a streaming JSON parser that lets you inspect the key name and string being parsed can handle this. Case solved and you can also redesign your code so you are not creating a temporary object every single parse that you most likely copy into actual objects later.

Not serializing BigInt is questionable to me but even that can be solved in userland.

On Saturday, July 14, 2018, Anders Rundgren <[hidden email]> wrote:
var small = BigInt("5");
var big = BigInt("5555555555555555555555555500003");
JSON.stringify([big,small]);
VM330:1 Uncaught TypeError: Do not know how to serialize a BigInt
    at JSON.stringify (<anonymous>)
    at <anonymous>:1:6

JSON Number serialization has apparently reached a new level (of confusion).

Personally I don't see the problem.  XML did just fine without hard-coded data types.

The JSON type system is basically a relic from JavaScript.  As such it has proved to be quite useful.
However, when you are outside of that scope, the point with the JSON type system gets pretty much zero since you anyway need to map extended types.

Oracle's JSON-B solution which serializes small values as Number and large values as String rather than having a unified serialization based on the underlying data type seems like a pretty broken concept although indeed fully conforming to the JSON specification. "Like the Devil reads the Bible" as we say in Scandinavia :-)

Adding a couple of double quotes is a major problem?  If so, it seems like a way more useful project making quotes optional for keys (named in a specific way), like they already are in JavaScript.

Yeah, and of course adding support for comments.

Anders

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON support for BigInt in Chrome/V8

Anders Rundgren-2
On 2018-07-28 21:21, Ranando King wrote:
> What I meant by complete is that if the value isn't an ES Number, isn't a boolean, and isn't null, it will have a string representation, or a binary representation that can be encoded as a string using base64. By using a dataUri for those cases, we can create a consistent library to handle those issues automatically, and without guesswork (no magic). If such a utility API became a standard in ES, other languages would be far more likely to adopt the standardized notation that would be provided.

As I have tried (apparently in vain) to describe, the other languages already have solutions for BigInt (and much more), which the respectively authors consider both sufficient and good.

ES6 is pretty late in the game and therefore rather have to adapt itself to the other players.

These are the options:
https://esdiscuss.org/topic/json-support-for-bigint-in-chrome-v8#content-30

- "Conservative" works right out of the box (although in a somewhat inelegant way).  It is backward compatible with many existing ES and non-ES JSON based applications.

- "RFC8259" support would require changes to the ES6 JSON object.  For parsing (`JSON.parse2()`), a `JSONNumber` object as proposed by Michael Theriot seems like a candidate.

I have limited faith in extending the (for general purpose information interchange) already pretty deficient JSON type scheme; state-of-the-art applications check indata anyway making explicit type information redundant.

Anders

>
> On Sat, Jul 28, 2018 at 12:03 PM Michael Theriot <[hidden email] <mailto:[hidden email]>> wrote:
>
>     I think we should move away from guessing and automagically parsing our JSON. The fact is a JSON number != JS number. Perhaps in userland (or a proposal...) JSON.parse could return with a JSONNumber object which the developer decides to convert to BigInt or Number.
>
>     On Saturday, July 28, 2018, J Decker <[hidden email] <mailto:[hidden email]>> wrote:
>
>
>
>         On Sat, Jul 28, 2018 at 6:57 AM Michael Theriot <[hidden email] <mailto:[hidden email]>> wrote:
>
>             In a language with arbitrary integer precision, Python 3 for example, the way to parse a "BigInt" would just be a plain, human readable number without quotes. The way to serialize it is the same. Any other kind of representation is out of spec, a workaround, and belongs in userland.
>
>
>         The problem with this, 'guessing' whether is't a Number() or a BigInt() is that numbers and bigint's don't interop.
>
>         {a:123, b:123n}
>         " { "a":123, "b":123 }"  any expressions that expect B to be a BigInt() will fail, becasue it will be in an expression of other bigints.
>
>         bigInts aren't just a better Number type, but, rather require other bigints for their expressions.
>
>
>             I think BigInt should serialize the same, not as strings or anything that is not a number. JSON.parse being unable to parse back into BigInt is a separate issue. It is solvable by using better parsing methods, not the convenient built-in one which has other issues. E.g. a streaming JSON parser that lets you inspect the key name and string being parsed can handle this. Case solved and you can also redesign your code so you are not creating a temporary object every single parse that you most likely copy into actual objects later.
>
>             Not serializing BigInt is questionable to me but even that can be solved in userland.
>
>             On Saturday, July 14, 2018, Anders Rundgren <[hidden email] <mailto:[hidden email]>> wrote:
>
>                 var small = BigInt("5");
>                 var big = BigInt("5555555555555555555555555500003");
>                 JSON.stringify([big,small]);
>                 VM330:1 Uncaught TypeError: Do not know how to serialize a BigInt
>                      at JSON.stringify (<anonymous>)
>                      at <anonymous>:1:6
>
>                 JSON Number serialization has apparently reached a new level (of confusion).
>
>                 Personally I don't see the problem.  XML did just fine without hard-coded data types.
>
>                 The JSON type system is basically a relic from JavaScript.  As such it has proved to be quite useful.
>                 However, when you are outside of that scope, the point with the JSON type system gets pretty much zero since you anyway need to map extended types.
>
>                 Oracle's JSON-B solution which serializes small values as Number and large values as String rather than having a unified serialization based on the underlying data type seems like a pretty broken concept although indeed fully conforming to the JSON specification. "Like the Devil reads the Bible" as we say in Scandinavia :-)
>
>                 Adding a couple of double quotes is a major problem?  If so, it seems like a way more useful project making quotes optional for keys (named in a specific way), like they already are in JavaScript.
>
>                 Yeah, and of course adding support for comments.
>
>                 Anders
>
>                 _______________________________________________
>                 es-discuss mailing list
>                 [hidden email] <mailto:[hidden email]>
>                 https://mail.mozilla.org/listinfo/es-discuss
>
>             _______________________________________________
>             es-discuss mailing list
>             [hidden email] <mailto:[hidden email]>
>             https://mail.mozilla.org/listinfo/es-discuss
>
>     _______________________________________________
>     es-discuss mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.mozilla.org/listinfo/es-discuss
>
>
>
> _______________________________________________
> es-discuss mailing list
> [hidden email]
> https://mail.mozilla.org/listinfo/es-discuss
>

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON support for BigInt in Chrome/V8

kai zhu
i support @anders’ *conservative* scheme and opened up a github-issue to try and get it included before stage-4 [1].  this might be one of the last-opportunities to change the behavior before its set-in-stone.

there was a previous github-issue on bigint JSON-support [2] (that led to the current behavior no one is happy about), but it never discussed the *conservative* scheme, hence the revisit.

to summarize, the *conservative* scheme basically copies the same JSON-behavior as Date.  users are already familiar with serializing Date, so there shouldn’t be any surprises (or at least less-surprises than throwing an error).  this is the simplest, actionable, and least-offensive-to-all-parties solution i can think of (unless someone has a less-offensive-to-all idea).  its also applicable in solving the same-issue for future primitives like BigDecimal.

```js
var aa;

// we copy JSON-behavior of Date
aa = 12345678901234567890n // <bigint primitive>
aa = JSON.stringify(aa) // '"12345678901234567890"' (escaped string)
aa = JSON.parse(aa) // '12345678901234567890' (un-escaped string)
aa = BigInt(aa) // <bigint primitive> (no precision-loss)

aa = new Date() // <Date object>
aa = JSON.stringify(aa) // '"2018-07-28T09:41:47.519Z"'' (escaped string)
aa = JSON.parse(aa) // '2018-07-28T09:41:47.519Z' (un-escaped string)
aa = new Date(aa) // <Date object>
```

[1] github-issue - revisit: Support JSON serialisation of BigInt values
https://github.com/tc39/proposal-bigint/issues/162

[2] github-issue - [closed] Support JSON serialisation of BigInt values


On 29 Jul 2018, at 12:17 PM, Anders Rundgren <[hidden email]> wrote:

As I have tried (apparently in vain) to describe, the other languages already have solutions for BigInt (and much more), which the respectively authors consider both sufficient and good.

ES6 is pretty late in the game and therefore rather have to adapt itself to the other players.

These are the options:
https://esdiscuss.org/topic/json-support-for-bigint-in-chrome-v8#content-30

- "Conservative" works right out of the box (although in a somewhat inelegant way).  It is backward compatible with many existing ES and non-ES JSON based applications.

- "RFC8259" support would require changes to the ES6 JSON object.  For parsing (`JSON.parse2()`), a `JSONNumber` object as proposed by Michael Theriot seems like a candidate.

I have limited faith in extending the (for general purpose information interchange) already pretty deficient JSON type scheme; state-of-the-art applications check indata anyway making explicit type information redundant.

Anders


_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON support for BigInt in Chrome/V8

T.J. Crowder-2
On Sun, Jul 29, 2018 at 9:01 AM, kai zhu
<[hidden email]> wrote:
> i support @anders’ *conservative* scheme and opened up a github-issue to try
> and get it included before stage-4 [1].

I agree with you and Anders that this should be sorted out now, not as a follow-on proposal. If it's left to later, people will supply their own `BigInt.prototype.toJSON` and cause themselves compatibility problems down-the-line.

Don't like the raw number in quotes ("conservative" option) at all. There should be *some* indication that this is a bigint, just as pattern matching tells us the default Date serialization is a date. This could be achieved with just the lower-case `n` at the end, as in some early examples in the github issue. And to support easy two-way, `BigInt(string)` should support the lower-case `n` at the end.

```js
aa = 12345678901234567890n; // BigInt primitive
aa = JSON.stringify(aa);    // '"12345678901234567890n"'
aa = JSON.parse(aa);        // '12345678901234567890n'
aa = BigInt(aa);            // BigInt primitive
```

`^\d+n$` isn't much of a pattern, but it's a pattern. (Earlier in this thread I suggested `/BitInt(123)/` to match Microsoft's `/Date(123)/` which is unambiguous and offers a path to extendibility, but Date already goes a different way...)

This parallels Date handling and seems a reasonable stop-gap until the next thing after JSON.

-- T.J. Crowder

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: JSON support for BigInt in Chrome/V8

Rob Ede
In reply to this post by Anders Rundgren-2
+1 on getting this sorted before stage 4

As people have said before, JSON already supports BigInts. We of course need to preserve backwards compatibility with JS and try not to break how other languages use JSON.

I think the best way to satisfy everyone is to give ourselves some flexibility going forwards and JSONifying BigInts as strings doesn’t allow JSON to do what it *can* do. That and it breaks the idea that JSON.parse(JSON.stringify(x)) is a deep clone.

So, flexibility: there are two options in my head for this.

1. An object passed to JSON.parse with options for how to deal with integers larger than Number can support. The current behaviour of misrepresenting them is the default. But other options could include converting to BigInts (or strings) if needed and always converting to BigInts (since they are non-interoperable currently).
This approach allows the programmer to decide the behavior and plan for it in their code. 

2. It seems like the already suggested JSONNumber object could offer the same flexibility and guarantees though. A `valueOf` that returns the misrepresented Number for interop with existing numbers. A `toString` that gives a well represented string conversion. (Im not 100% certain of the order these two functions are used in auto-casting*. On mobile or else would check spec.) A `toBigInt` / `unwrap` (modad-ish) terminology to get the true value of and when the programmer wants.

Hope these ideas seem reasonable. Scrutiny welcome.

Regards,
Rob J T Ede


On 29 Jul 2018, at 10:04, T.J. Crowder <[hidden email]> wrote:

On Sun, Jul 29, 2018 at 9:01 AM, kai zhu
<[hidden email]> wrote:
> i support @anders’ *conservative* scheme and opened up a github-issue to try
> and get it included before stage-4 [1].

I agree with you and Anders that this should be sorted out now, not as a follow-on proposal. If it's left to later, people will supply their own `BigInt.prototype.toJSON` and cause themselves compatibility problems down-the-line.

Don't like the raw number in quotes ("conservative" option) at all. There should be *some* indication that this is a bigint, just as pattern matching tells us the default Date serialization is a date. This could be achieved with just the lower-case `n` at the end, as in some early examples in the github issue. And to support easy two-way, `BigInt(string)` should support the lower-case `n` at the end.

```js
aa = 12345678901234567890n; // BigInt primitive
aa = JSON.stringify(aa);    // '"12345678901234567890n"'
aa = JSON.parse(aa);        // '12345678901234567890n'
aa = BigInt(aa);            // BigInt primitive
```

`^\d+n$` isn't much of a pattern, but it's a pattern. (Earlier in this thread I suggested `/BitInt(123)/` to match Microsoft's `/Date(123)/` which is unambiguous and offers a path to extendibility, but Date already goes a different way...)

This parallels Date handling and seems a reasonable stop-gap until the next thing after JSON.

-- T.J. Crowder

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
123