Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Claude Pache

TL;DR: We propose to add functionalities to the `for/of` loop that make all features of generators (and iterators) available. Currently, the features unavailable in `for/of` loops are: values sent to the generator that are retrieved by the `yield` expression, and closing value produced by the generator using the `return` statement. The pretext is to make it possible to rewrite naturally `someArray.reduce(functionExpression)` et al. when used inside a generator function, so that `functionExpression` could contain `yield` statements, a problem raised in a recent thread.


In a recent thread [1], it has been noted that, when using `.forEach` and friends inside a generator, you cannot use `yield` inside the callback.

There has been some reflections on how to fix generators in order to avoid that defect; but it was taking the problem by the wrong side, for the issue does not come from the generator function,
but from the `.forEach` construct: it uses a first-class callback function where `for/of` uses a second-class block.

Let's see how to reimplement `.forEach` et al. with ES6 tools, so that it just works. Consider a generic loop:

```
    // `arr` is an array
    for (let v of arr) {
        // do something with with v
    }
```

Suppose that you want to run the loop only for even-ranked items of the array. You could do something like this:

```
    Array.prototype.forEachEven = function(f, o) {
        // For simplicity, we ignore the issue of sparse arrays.
        for (let i = 0; i < this.length; i += 2) {
            f.call(o, this[i], i, this)
        }
    }

    arr.forEachEven(function(v) {
        // do something with v
        // But you can't use yield, break or return :-(
    }, this)
```

However, instead of transforming the entire for/of loop, it is more on-focus to act on the iteration part only. I mean the following:

```
    function* gEven(iterable) {
        let i = 0
        for (let x of iterable) {
            if (i % 2 === 0) {
                yield x
            }
            i += 1
        }
    }

    for (let v of gEven(arr)) {
        // do something with with v
        // You can use yield, break or return here, and it will just work :-)
    }
```

As you see, no more callback, no more problem. (For sure, you can *also* use `yield*` inside the loop in order to abstract away some code; but the focus of this message is on the iteration protocol.)

Now (and from here the things are becoming interesting), let us try to reimplement `Array.prototype.some` and `Array.prototype.reduce`
using that pattern:

```
    r = arr.some(function(v) {
        let condition
        // compute `condition`
        return condition
    }, this)

    r = arr.reduce(function(x, v) {
        let combined
        // combine `x` and `v` into `combined`
        return combined
    }, initialValue)
```

Here, we need:
(1) retrieve the intermediate results returned by the callback (`condition` and `combined` in our examples);
(2) provide the final result of the loop (the value that'll be stored in `r`)

On the side of generators, all is already done, for you can:
(1) retrieve an intermediate result by evaluating a `yield` expression;
(2) produce the final result using a `return` statement.
or, more generally, in the case of iterators:
(1) an intermediate result is retrieved by the first argument of the `next` method`;
(2) the final result is produced by returning `{ done: true, value: result }`.

Thus, we obtain:
```
    function* gSome(iterator) {
        for (let v of iterator) {
            if (yield v)
                return true
        }
        return false
    }

    function* gReduce(iterator, r) {
        for (let v of iterator) {
            r = yield [r, v]
        }
        return r
    }
```

However, on the side of the `for/of loop`, there is currently (to my knowledge) no way to send or retrieve results to/from the iterator being traversed.
Therefore, I propose to add the functionality:

(1) Sending an intermediate result to the iterator: use a `continue with expression` statement.
(2) Producing the final result: The for/of loop becomes an expression, which evaluates to either:
    (a) the closing value sent by the iterator, or:
    (b) the `expression` of a `break with expression` statement.

(There is no LineTerminator between `continue/break` and `with`. The use of the reserved word `with` means that it can't be confused with a label.)
Using that syntax, our loops are written as following:

```
    r = for (let v of gSome(arr)) {
        let condition
        // compute `condition`, using `yield` ad libitum
        continue with condition
    }

    r = for (let [x, v] of gReduce(arr, initialValue)) {
        let combined
        // combine `x` and `v` into `combined`, using `yield` ad libitum
        continue with combined
    }

    r = for (let x of itr) {
        break with 42
    }
    // `r` will be equal to 42 if `itr` is not empty
```

Naturally, you can also use labels:
```
    break label with expression;
    continue label with expression;
```

It is true that the examples chosen are academic in the sense it isn't harder to write:
```
    var r = initialValue
    for (let v of arr) {
        let combined
        // combine `r` and `v` into `combined`, using `yield` ad libitum
        r = combined
    }
```

Netherveless, it is a shame that some nice features of iteration protocol are unavailable in `for/of` loops, which is probably intended to be the main consumer of iterators.

(And, if nothing else, it is a nice opportunity for the `with` keyword to redeem itself. :-P)

—Claude

[1] generators vs forEach: https://mail.mozilla.org/pipermail/es-discuss/2013-July/031919.html


_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Andrew Fedoniouk
Slightly different approach that I've found [1] quite useful.

Consider this simple function:

function range(start,end) {
       var i = start;
       return function() {
          if( i < end ) return i++;
       }
}

It returns so called iterator function (not an Iterator but just a function).

and imagine that for/in ( or for/of )

   for(var el in something)

statement is modified to support functions as collections. So body of
the for loop is called until function
`something` returns anything but not void.

Having this this we can write something like this:

   for(var i in range(12,48))  {
       console.log(i);
  }

In principle this simple change eliminates need of most of
generator/yield cases.

I am not sure if it close to the problem you describe but something
tells me that it is.

[1] http://www.codeproject.com/Articles/33662/TIScript-language-a-gentle-extension-of-JavaScript


On Tue, Jul 30, 2013 at 5:21 AM, Claude Pache <[hidden email]> wrote:

>
> TL;DR: We propose to add functionalities to the `for/of` loop that make all features of generators (and iterators) available. Currently, the features unavailable in `for/of` loops are: values sent to the generator that are retrieved by the `yield` expression, and closing value produced by the generator using the `return` statement. The pretext is to make it possible to rewrite naturally `someArray.reduce(functionExpression)` et al. when used inside a generator function, so that `functionExpression` could contain `yield` statements, a problem raised in a recent thread.
>
>
> In a recent thread [1], it has been noted that, when using `.forEach` and friends inside a generator, you cannot use `yield` inside the callback.
>
> There has been some reflections on how to fix generators in order to avoid that defect; but it was taking the problem by the wrong side, for the issue does not come from the generator function,
> but from the `.forEach` construct: it uses a first-class callback function where `for/of` uses a second-class block.
>
> Let's see how to reimplement `.forEach` et al. with ES6 tools, so that it just works. Consider a generic loop:
>
> ```
>     // `arr` is an array
>     for (let v of arr) {
>         // do something with with v
>     }
> ```
>
> Suppose that you want to run the loop only for even-ranked items of the array. You could do something like this:
>
> ```
>     Array.prototype.forEachEven = function(f, o) {
>         // For simplicity, we ignore the issue of sparse arrays.
>         for (let i = 0; i < this.length; i += 2) {
>             f.call(o, this[i], i, this)
>         }
>     }
>
>     arr.forEachEven(function(v) {
>         // do something with v
>         // But you can't use yield, break or return :-(
>     }, this)
> ```
>
> However, instead of transforming the entire for/of loop, it is more on-focus to act on the iteration part only. I mean the following:
>
> ```
>     function* gEven(iterable) {
>         let i = 0
>         for (let x of iterable) {
>             if (i % 2 === 0) {
>                 yield x
>             }
>             i += 1
>         }
>     }
>
>     for (let v of gEven(arr)) {
>         // do something with with v
>         // You can use yield, break or return here, and it will just work :-)
>     }
> ```
>
> As you see, no more callback, no more problem. (For sure, you can *also* use `yield*` inside the loop in order to abstract away some code; but the focus of this message is on the iteration protocol.)
>
> Now (and from here the things are becoming interesting), let us try to reimplement `Array.prototype.some` and `Array.prototype.reduce`
> using that pattern:
>
> ```
>     r = arr.some(function(v) {
>         let condition
>         // compute `condition`
>         return condition
>     }, this)
>
>     r = arr.reduce(function(x, v) {
>         let combined
>         // combine `x` and `v` into `combined`
>         return combined
>     }, initialValue)
> ```
>
> Here, we need:
> (1) retrieve the intermediate results returned by the callback (`condition` and `combined` in our examples);
> (2) provide the final result of the loop (the value that'll be stored in `r`)
>
> On the side of generators, all is already done, for you can:
> (1) retrieve an intermediate result by evaluating a `yield` expression;
> (2) produce the final result using a `return` statement.
> or, more generally, in the case of iterators:
> (1) an intermediate result is retrieved by the first argument of the `next` method`;
> (2) the final result is produced by returning `{ done: true, value: result }`.
>
> Thus, we obtain:
> ```
>     function* gSome(iterator) {
>         for (let v of iterator) {
>             if (yield v)
>                 return true
>         }
>         return false
>     }
>
>     function* gReduce(iterator, r) {
>         for (let v of iterator) {
>             r = yield [r, v]
>         }
>         return r
>     }
> ```
>
> However, on the side of the `for/of loop`, there is currently (to my knowledge) no way to send or retrieve results to/from the iterator being traversed.
> Therefore, I propose to add the functionality:
>
> (1) Sending an intermediate result to the iterator: use a `continue with expression` statement.
> (2) Producing the final result: The for/of loop becomes an expression, which evaluates to either:
>     (a) the closing value sent by the iterator, or:
>     (b) the `expression` of a `break with expression` statement.
>
> (There is no LineTerminator between `continue/break` and `with`. The use of the reserved word `with` means that it can't be confused with a label.)
> Using that syntax, our loops are written as following:
>
> ```
>     r = for (let v of gSome(arr)) {
>         let condition
>         // compute `condition`, using `yield` ad libitum
>         continue with condition
>     }
>
>     r = for (let [x, v] of gReduce(arr, initialValue)) {
>         let combined
>         // combine `x` and `v` into `combined`, using `yield` ad libitum
>         continue with combined
>     }
>
>     r = for (let x of itr) {
>         break with 42
>     }
>     // `r` will be equal to 42 if `itr` is not empty
> ```
>
> Naturally, you can also use labels:
> ```
>     break label with expression;
>     continue label with expression;
> ```
>
> It is true that the examples chosen are academic in the sense it isn't harder to write:
> ```
>     var r = initialValue
>     for (let v of arr) {
>         let combined
>         // combine `r` and `v` into `combined`, using `yield` ad libitum
>         r = combined
>     }
> ```
>
> Netherveless, it is a shame that some nice features of iteration protocol are unavailable in `for/of` loops, which is probably intended to be the main consumer of iterators.
>
> (And, if nothing else, it is a nice opportunity for the `with` keyword to redeem itself. :-P)
>
> —Claude
>
> [1] generators vs forEach: https://mail.mozilla.org/pipermail/es-discuss/2013-July/031919.html
>
>
> _______________________________________________
> es-discuss mailing list
> [hidden email]
> https://mail.mozilla.org/listinfo/es-discuss



--
Andrew Fedoniouk.

http://terrainformatica.com
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Rick Waldron



On Tue, Jul 30, 2013 at 3:39 PM, Andrew Fedoniouk <[hidden email]> wrote:
Slightly different approach that I've found [1] quite useful.

Consider this simple function:

function range(start,end) {
       var i = start;
       return function() {
          if( i < end ) return i++;
       }
}

It returns so called iterator function (not an Iterator but just a function).

and imagine that for/in ( or for/of )

   for(var el in something)

statement is modified to support functions as collections. So body of
the for loop is called until function
`something` returns anything but not void. 

Having this this we can write something like this:

   for(var i in range(12,48))  {
       console.log(i);
  }

In principle this simple change eliminates need of most of
generator/yield cases.


Which is exactly this:

function * range(start, end) {
  for (var i = start; i <= end; i++) {
    yield i;
  }
}

for (var value of range(0, 9)) {
  console.log( value );
}


Or do you mean to say that generator functions and yield should be removed?


Rick
 

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Andrew Fedoniouk
On Tue, Jul 30, 2013 at 1:43 PM, Rick Waldron <[hidden email]> wrote:

>
>
>
> On Tue, Jul 30, 2013 at 3:39 PM, Andrew Fedoniouk
> <[hidden email]> wrote:
>>
>> Slightly different approach that I've found [1] quite useful.
>>
>> Consider this simple function:
>>
>> function range(start,end) {
>>        var i = start;
>>        return function() {
>>           if( i < end ) return i++;
>>        }
>> }
>>
>> It returns so called iterator function (not an Iterator but just a
>> function).
>>
>> and imagine that for/in ( or for/of )
>>
>>    for(var el in something)
>>
>> statement is modified to support functions as collections. So body of
>> the for loop is called until function
>> `something` returns anything but not void.
>>
>>
>> Having this this we can write something like this:
>>
>>    for(var i in range(12,48))  {
>>        console.log(i);
>>   }
>>
>> In principle this simple change eliminates need of most of
>> generator/yield cases.
>
>
>
> Which is exactly this:
>
> function * range(start, end) {
>   for (var i = start; i <= end; i++) {
>     yield i;
>   }
> }
>
> for (var value of range(0, 9)) {
>   console.log( value );
> }
>

Yes, it is close conceptually. With only exception: yield  requires stack
itself to be heap allocated thing. Many things around this.

In C++ I did very lightweight $generator/$yield implementation [1] but
unfortunately that trick is not available in JS.

>
> Or do you mean to say that generator functions and yield should be removed?
>

In principle, with functions-as-collections the yield and the whole
generators stuff
is not needed. If functions can be used on the right of 'in' or 'of'
in 'for' then
all 'yield' use cases that I saw so far can be implemented without the yield.
So why do we need redundant entities?


[1] http://www.codeproject.com/Articles/29524/Generators-in-C


--
Andrew Fedoniouk.

http://terrainformatica.com
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Tab Atkins Jr.
On Tue, Jul 30, 2013 at 2:10 PM, Andrew Fedoniouk
<[hidden email]> wrote:
> On Tue, Jul 30, 2013 at 1:43 PM, Rick Waldron <[hidden email]> wrote:
>> Or do you mean to say that generator functions and yield should be removed?
>
> In principle, with functions-as-collections the yield and the whole
> generators stuff
> is not needed. If functions can be used on the right of 'in' or 'of'
> in 'for' then
> all 'yield' use cases that I saw so far can be implemented without the yield.
> So why do we need redundant entities?

You don't understand what yield does.  It freezes the execution of the
function at that point, waiting until .next() is called to resume
execution.  This means you can do a number of very convenient things,
like yielding in the middle of a loop, or yielding within a try/catch
block.

You can't do these with simple functions without significant
refactoring.  The convenience of yield is well-established by Python.

~TJ
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Quildreen Motta
In reply to this post by Andrew Fedoniouk

In principle, with functions-as-collections the yield and the whole
generators stuff
is not needed. If functions can be used on the right of 'in' or 'of'
in 'for' then
all 'yield' use cases that I saw so far can be implemented without the yield.
So why do we need redundant entities?


_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Andrew Fedoniouk
In reply to this post by Tab Atkins Jr.
On Tue, Jul 30, 2013 at 2:14 PM, Tab Atkins Jr. <[hidden email]> wrote:

> On Tue, Jul 30, 2013 at 2:10 PM, Andrew Fedoniouk
> <[hidden email]> wrote:
>> On Tue, Jul 30, 2013 at 1:43 PM, Rick Waldron <[hidden email]> wrote:
>>> Or do you mean to say that generator functions and yield should be removed?
>>
>> In principle, with functions-as-collections the yield and the whole
>> generators stuff
>> is not needed. If functions can be used on the right of 'in' or 'of'
>> in 'for' then
>> all 'yield' use cases that I saw so far can be implemented without the yield.
>> So why do we need redundant entities?
>
> You don't understand what yield does.  It freezes the execution of the
> function at that point, waiting until .next() is called to resume
> execution.  This means you can do a number of very convenient things,
> like yielding in the middle of a loop, or yielding within a try/catch
> block.

Trust me, I do understand what yield does. Check the link I've
provided - it implements
in C++ just what you said.


OK, here is an example that has multiple yielding exits:

function fruits() {
  var state = 0;
  return function() {
    switch(state++) {
      case 0: return "apple";
      case 1: return "orange";
      case 2: return "lime";
    }
  }
}

So if you will run this:

for(var fr in fruits)
  console.log( fr )

you will get
 apple
 orange
 lime

printed out.

Close enough to what you will do with yield.

*Any* yield use case can be reproduced this way.

>
> You can't do these with simple functions without significant
> refactoring.  The convenience of yield is well-established by Python.
>

Yep, but don't forget that Python uses reference counting.
In Python generator creation and destruction is deterministic -
you create it at the entry of for loop and it gets destroyed (or at
least can be)
at the end of it.  In JS the only option is to make stack GC-able thing -
generator gets created and live until the GC cycle.

Is this acceptable price for the feature - I really don't know.
But I do know that for simple enumerators/iterators cases
plain functions are more optimal in JS case. And they do not require
such quite ugly syntax changes as "function*".


--
Andrew Fedoniouk.

http://terrainformatica.com
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Claude Pache
In reply to this post by Andrew Fedoniouk

Le 30 juil. 2013 à 21:39, Andrew Fedoniouk <[hidden email]> a écrit :

>
> I am not sure if it close to the problem you describe but something
> tells me that it is.

You missed my point. I do not want to change the approach to iteration, even if it solves a problem. I am proposing additions around the for/of loop that exploits more completely the iteration protocol proposed for ES6, which has already matured lengthily (so I won't change it). The problem I've taken is merely an illustration for the concept.

—Claude
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss