Generic piping utilities.

Other functions can be piped to them, from both sides, without having to use the pipe object. The resulting function then also inherits this functionality.


Even though these are defined in the pipetools.utils module, they’re importable directly from pipetools for convenience.

All of these that take a function as an argument can automatically partially apply the given function with positional and/or keyword arguments, for example:

foreach(some_func, foo, bar=None)

Is the same as:

foreach(partial(some_func, foo, bar=None))

(As of 0.1.9 this uses X-partial)

They also automatically convert the X object to an actual function.

List of built-in utils

pipetools.utils.as_args(function, *args, **kwargs)

Applies the sequence in the input as positional arguments to function.

some_lists > as_args(izip)
pipetools.utils.as_kwargs(function, *args, **kwargs)

Applies the dictionary in the input as keyword arguments to function.

pipetools.utils.debug_print(function, *args, **kwargs)

Prints function applied on input and returns the input.

foo = (pipe
    | something
    | debug_print(X.get_status())
    | something_else
    | foreach(debug_print("attr is: {0.attr}"))
    | etc)

Assumes an iterable on the input, returns an iterable with identical items except for the first count.

>>> xrange(10) > drop_first(5) | tuple
(5, 6, 7, 8, 9)
pipetools.utils.foreach(function, *args, **kwargs)

Returns a function that takes an iterable and returns an iterator over the results of calling function on each item of the iterable.

>>> xrange(5) > foreach(factorial) | list
[1, 1, 2, 6, 24]
pipetools.utils.foreach_do(function, *args, **kwargs)

Like foreach() but is evaluated immediately and doesn’t return anything.

For the occasion that you just want to do some side-effects:

open('addresses.txt') > foreach(geocode) | foreach_do(launch_missile)

– With foreach() nothing would happen (except an itetrator being created)

pipetools.utils.group_by(function, *args, **kwargs)

Groups input sequence by function.

Returns an iterator over a sequence of tuples where the first item is a result of function and the second one a list of items matching this result.

Ordering of the resulting iterator is undefined, but ordering of the items in the groups is preserved.

>>> [1, 2, 3, 4, 5, 6] > group_by(X % 2) | list
[(0, [2, 4, 6]), (1, [1, 3, 5])]
pipetools.utils.select_first(function, *args, **kwargs)

Returns first item from input sequence that satisfies condition. Or None if none does.

>>> ['py', 'pie', 'pi'] > select_first(X.startswith('pi'))

As of 0.2.1 you can also directly use regular expressions and write the above as:

>>> ['py', 'pie', 'pi'] > select_first('^pi')

There is also a shortcut for select_first(X) called first_of:

>>> first_of(['', None, 0, 3, 'something'])
>>> first_of([])
pipetools.utils.sort_by(function, *args, **kwargs)

Sorts an incoming sequence by using the given function as key.

>>> xrange(10) > sort_by(-X)
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

Supports automatic data-structure creation:

users > sort_by([X.last_name, X.first_name])

There is also a shortcut for sort_by(X) called sort:

>>> [4, 5, 8, -3, 0] > sort
[-3, 0, 4, 5, 8]

And (as of 0.2.3) a shortcut for reversing the sort:

>>> 'asdfaSfa' > sort_by(X.lower()).descending
['s', 'S', 'f', 'f', 'd', 'a', 'a', 'a']

Assumes an iterable on the input, returns an iterable with first count items from the input (or possibly less, if there isn’t that many).

>>> xrange(9000) > where(X % 100 == 0) | take_first(5) | tuple
(0, 100, 200, 300, 400)
pipetools.utils.take_until(function, *args, **kwargs)
>>> [1, 4, 6, 4, 1] > take_until(X > 5) | list
[1, 4]
pipetools.utils.unless(exception_class_or_tuple, func, *args, **kwargs)

When exception_class_or_tuple occurs while executing func, it will be caught and None will be returned.

>>> f = where(X > 10) | list | unless(IndexError, X[0])
>>> f([5, 8, 12, 4])
>>> f([1, 2, 3])
pipetools.utils.where(function, *args, **kwargs)

Pipe-able lazy filter.

>>> odd_range = xrange | where(X % 2) | list
>>> odd_range(10)
[1, 3, 5, 7, 9]
pipetools.utils.where_not(function, *args, **kwargs)

Inverted where().

Make your own

You can make your own, but you generally shouldn’t need to, since you can pipe any functions you like.

But if you feel like there’s a generally reusable pipe-util missing, feel free to add it via a pull request on github.

How to do it? Just write the function and stick the pipe_util() decorator on it.

And optionally also auto_string_formatter() (see Automatic string formatting) or data_structure_builder() (see Automatic data-structure creation) if it makes sense.

Automatic data-structure creation

Some of the utils, most importantly foreach(), offer a shortcut for creating basic python data structures - list, tuple and dict.

It works like this (the | tuple at the end is just so we can see the result, otherwise it would just give us <iterable thing at 0xasdf123>):

>>> xrange(5) > foreach({X: X * 2}) | tuple
({0: 0}, {1: 2}, {2: 4}, {3: 6}, {4: 8})

>>> xrange(5) > foreach([X, X * u'★']) | tuple
([0, u''], [1, u'★'], [2, u'★★'], [3, u'★★★'], [4, u'★★★★'])

It can also be combined with string formatting:

>>> names = [('John', 'Matrix'), ('Jack', 'Slater')]
>>> names > foreach({'name': "{0} {1}", 'initials': '{0[0]}. {1[0]}.'}) | tuple
({u'initials': u'J. M.', u'name': u'John Matrix'},
 {u'initials': u'J. S.', u'name': u'Jack Slater'})

Automatic regex conditions

If you use a string instead of a function as a condition in where(), where_not(), select_first() or take_until() the string will be used as a regex to match the input against. This will, of course, work only if the items of the input sequence are strings.



is equivalent to:

where(re.match, r'^some\-regexp?$')

If you want to easily add this functionality to your own functions, you can use the regex_condition() decorator.

Table Of Contents

Previous topic

Welcome to pipetools’ documentation!

Next topic

The X object