pipe-utils

Generic piping utilities.

Other functions can be piped to them, from both sides, without having to use the pipe object. The resulting function then also inherits this functionality.

Built-in

Even though these are defined in the pipetools.utils module, they’re importable directly from pipetools for convenience.

All of these that take a function as an argument can automatically partially apply the given function with positional and/or keyword arguments, for example:

foreach(some_func, foo, bar=None)

Is the same as:

foreach(partial(some_func, foo, bar=None))

(As of 0.1.9 this uses X-partial)

They also automatically convert the X object to an actual function.

List of built-in utils

pipetools.utils.as_args(function, *args, **kwargs)

Applies the sequence in the input as positional arguments to function.

some_lists > as_args(izip)
pipetools.utils.as_kwargs(function, *args, **kwargs)

Applies the dictionary in the input as keyword arguments to function.

pipetools.utils.debug_print(function, *args, **kwargs)

Prints function applied on input and returns the input.

foo = (pipe
    | something
    | debug_print(X.get_status())
    | something_else
    | foreach(debug_print("attr is: {0.attr}"))
    | etc)
pipetools.utils.drop_first(count)

Assumes an iterable on the input, returns an iterable with identical items except for the first count.

>>> xrange(10) > drop_first(5) | tuple
(5, 6, 7, 8, 9)
pipetools.utils.foreach(function, *args, **kwargs)

Returns a function that takes an iterable and returns an iterator over the results of calling function on each item of the iterable.

>>> xrange(5) > foreach(factorial) | list
[1, 1, 2, 6, 24]
pipetools.utils.foreach_do(function, *args, **kwargs)

Like foreach() but is evaluated immediately and doesn’t return anything.

For the occasion that you just want to do some side-effects:

open('addresses.txt') > foreach(geocode) | foreach_do(launch_missile)

– With foreach() nothing would happen (except an itetrator being created)

pipetools.utils.group_by(function, *args, **kwargs)

Groups input sequence by function.

Returns an iterator over a sequence of tuples where the first item is a result of function and the second one a list of items matching this result.

Ordering of the resulting iterator is undefined, but ordering of the items in the groups is preserved.

>>> [1, 2, 3, 4, 5, 6] > group_by(X % 2) | list
[(0, [2, 4, 6]), (1, [1, 3, 5])]
pipetools.utils.select_first(function, *args, **kwargs)

Returns first item from input sequence that satisfies condition. Or None if none does.

>>> ['py', 'pie', 'pi'] > select_first(X.startswith('pi'))
'pie'

As of 0.2.1 you can also directly use regular expressions and write the above as:

>>> ['py', 'pie', 'pi'] > select_first('^pi')
'pie'

There is also a shortcut for select_first(X) called first_of:

>>> first_of(['', None, 0, 3, 'something'])
3
>>> first_of([])
None
pipetools.utils.sort_by(function, *args, **kwargs)

Sorts an incoming sequence by using the given function as key.

>>> xrange(10) > sort_by(-X)
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

Supports automatic data-structure creation:

users > sort_by([X.last_name, X.first_name])

There is also a shortcut for sort_by(X) called sort:

>>> [4, 5, 8, -3, 0] > sort
[-3, 0, 4, 5, 8]

And (as of 0.2.3) a shortcut for reversing the sort:

>>> 'asdfaSfa' > sort_by(X.lower()).descending
['s', 'S', 'f', 'f', 'd', 'a', 'a', 'a']
pipetools.utils.take_first(count)

Assumes an iterable on the input, returns an iterable with first count items from the input (or possibly less, if there isn’t that many).

>>> xrange(9000) > where(X % 100 == 0) | take_first(5) | tuple
(0, 100, 200, 300, 400)
pipetools.utils.take_until(function, *args, **kwargs)
>>> [1, 4, 6, 4, 1] > take_until(X > 5) | list
[1, 4]
pipetools.utils.unless(exception_class_or_tuple, func, *args, **kwargs)

When exception_class_or_tuple occurs while executing func, it will be caught and None will be returned.

>>> f = where(X > 10) | list | unless(IndexError, X[0])
>>> f([5, 8, 12, 4])
12
>>> f([1, 2, 3])
None
pipetools.utils.where(function, *args, **kwargs)

Pipe-able lazy filter.

>>> odd_range = xrange | where(X % 2) | list
>>> odd_range(10)
[1, 3, 5, 7, 9]
pipetools.utils.where_not(function, *args, **kwargs)

Inverted where().

Make your own

You can make your own, but you generally shouldn’t need to, since you can pipe any functions you like.

But if you feel like there’s a generally reusable pipe-util missing, feel free to add it via a pull request on github.

How to do it? Just write the function and stick the pipe_util() decorator on it.

And optionally also auto_string_formatter() (see Automatic string formatting) or data_structure_builder() (see Automatic data-structure creation) if it makes sense.

Automatic data-structure creation

Some of the utils, most importantly foreach(), offer a shortcut for creating basic python data structures - list, tuple and dict.

It works like this (the | tuple at the end is just so we can see the result, otherwise it would just give us <iterable thing at 0xasdf123>):

>>> xrange(5) > foreach({X: X * 2}) | tuple
({0: 0}, {1: 2}, {2: 4}, {3: 6}, {4: 8})

>>> xrange(5) > foreach([X, X * u'★']) | tuple
([0, u''], [1, u'★'], [2, u'★★'], [3, u'★★★'], [4, u'★★★★'])

It can also be combined with string formatting:

>>> names = [('John', 'Matrix'), ('Jack', 'Slater')]
>>> names > foreach({'name': "{0} {1}", 'initials': '{0[0]}. {1[0]}.'}) | tuple
({u'initials': u'J. M.', u'name': u'John Matrix'},
 {u'initials': u'J. S.', u'name': u'Jack Slater'})

Automatic regex conditions

If you use a string instead of a function as a condition in where(), where_not(), select_first() or take_until() the string will be used as a regex to match the input against. This will, of course, work only if the items of the input sequence are strings.

Essentially:

where(r'^some\-regexp?$')

is equivalent to:

where(re.match, r'^some\-regexp?$')

If you want to easily add this functionality to your own functions, you can use the regex_condition() decorator.

Table Of Contents

Previous topic

Welcome to pipetools’ documentation!

Next topic

The X object