pipe-utils

Generic piping utilities.

Other functions can be piped to them, from both sides, without having to use the pipe object. The resulting function then also inherits this functionality.

Built-in

Even though these are defined in the pipetools.utils module, they’re importable directly from pipetools for convenience.

All of these that take a function as an argument can automatically partially apply the given function with positional and/or keyword arguments, for example:

foreach(some_func, foo, bar=None)

Is the same as:

foreach(partial(some_func, foo, bar=None))

(As of 0.1.9 this uses X-partial)

They also automatically convert the X object to an actual function.

List of built-in utils

pipetools.utils.as_args(function)

Applies the sequence in the input as positional arguments to function.

some_lists > as_args(izip)

pipetools.utils.as_kwargs(function)

Applies the dictionary in the input as keyword arguments to function.

pipetools.utils.count

Returns the number of items in iterable.

pipetools.utils.debug_print(function)

Prints function applied on input and returns the input.

foo = (pipe
    | something
    | debug_print(X.get_status())
    | something_else
    | foreach(debug_print("attr is: {0.attr}"))
    | etc)

pipetools.utils.drop_first(count)

Assumes an iterable on the input, returns an iterable with identical items except for the first count.

>>> range(10) > drop_first(5) | tuple
(5, 6, 7, 8, 9)

pipetools.utils.flatten

Flattens an arbitrarily deep nested iterable(s).

>>> [[[[[[1]]], 2], range(2) > foreach(X + 3)]] > flatten | list
[1, 2, 3, 4]

Does not treat strings and (as of 0.3.1) mappings (dictionaries) as iterables so these are left alone.

>>> ('hello', [{'how': 'are'}, [['you']]]) > flatten | list
['hello', {'how': 'are'}, 'you']

Also turns non-iterables into iterables which is convenient when you are not sure about the input.

>>> 'stuff' > flatten | list
['stuff']

pipetools.utils.foreach(function)

Returns a function that takes an iterable and returns an iterator over the results of calling function on each item of the iterable.

>>> range(5) > foreach(factorial) | list
[1, 1, 2, 6, 24]

pipetools.utils.foreach_do(function)

Like foreach() but is evaluated immediately and doesn’t return anything.

For the occasion that you just want to do some side-effects:

open('addresses.txt') > foreach(geocode) | foreach_do(launch_missile)

– With foreach() nothing would happen (except an itetrator being created)

pipetools.utils.group_by(function)

Groups input sequence by function.

Returns an iterator over a sequence of tuples where the first item is a result of function and the second one a list of items matching this result.

Ordering of the resulting iterator is undefined, but ordering of the items in the groups is preserved.

>>> [1, 2, 3, 4, 5, 6] > group_by(X % 2) | list
[(0, [2, 4, 6]), (1, [1, 3, 5])]

pipetools.utils.select_first(condition)

Returns first item from input sequence that satisfies condition. Or None if none does.

>>> ['py', 'pie', 'pi'] > select_first(X.startswith('pi'))
'pie'

As of 0.2.1 you can also directly use regular expressions and write the above as:

>>> ['py', 'pie', 'pi'] > select_first('^pi')
'pie'

There is also a shortcut for select_first(X) called first_of:

>>> first_of(['', None, 0, 3, 'something'])
3
>>> first_of([])
None

pipetools.utils.sort_by(function)

Sorts an incoming sequence by using the given function as key.

>>> range(10) > sort_by(-X)
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

Supports automatic data-structure creation:

users > sort_by([X.last_name, X.first_name])

There is also a shortcut for sort_by(X) called sort:

>>> [4, 5, 8, -3, 0] > sort
[-3, 0, 4, 5, 8]

And (as of 0.2.3) a shortcut for reversing the sort:

>>> 'asdfaSfa' > sort_by(X.lower()).descending
['s', 'S', 'f', 'f', 'd', 'a', 'a', 'a']

pipetools.utils.take_first(count)

Assumes an iterable on the input, returns an iterable with first count items from the input (or possibly less, if there isn’t that many).

>>> range(9000) > where(X % 100 == 0) | take_first(5) | tuple
(0, 100, 200, 300, 400)

pipetools.utils.take_until(condition)

>>> [1, 4, 6, 4, 1] > take_until(X > 5) | list
[1, 4]

>>> [1, 4, 6, 4, 1] > take_until(X > 5).including | list
[1, 4, 6]

pipetools.utils.take_until_including(condition)

>>> [1, 4, 6, 4, 1] > take_until_including(X > 5) | list
[1, 4, 6]

pipetools.utils.tee(function)

Sends a copy of the input into function - like a T junction.

pipetools.utils.unless(exception_class_or_tuple, func, *args, **kwargs)

When exception_class_or_tuple occurs while executing func, it will be caught and None will be returned.

>>> f = where(X > 10) | list | unless(IndexError, X[0])
>>> f([5, 8, 12, 4])
12
>>> f([1, 2, 3])
None

pipetools.utils.where(condition)

Pipe-able lazy filter.

>>> odd_range = range | where(X % 2) | list
>>> odd_range(10)
[1, 3, 5, 7, 9]

pipetools.utils.where_not(condition)

Inverted where().

Make your own

You can make your own, but you generally shouldn’t need to, since you can pipe any functions you like.

But if you feel like there’s a generally reusable pipe-util missing, feel free to add it via a pull request on github.

How to do it? Just write the function and stick the pipe_util() decorator on it.

And optionally also auto_string_formatter() (see Automatic string formatting) or data_structure_builder() (see Automatic data-structure creation) if it makes sense.

Automatic data-structure creation

Some of the utils, most importantly foreach(), offer a shortcut for creating basic python data structures - list, tuple and dict.

It works like this (the | list at the end is just so we can see the result, otherwise it would just give us <iterable thing at 0xasdf123>):

>>> range(5) > foreach({X: X * 2}) | list
[{0: 0}, {1: 2}, {2: 4}, {3: 6}, {4: 8}]

>>> range(5) > foreach([X, X * '★']) | list
[[0, ''], [1, '★'], [2, '★★'], [3, '★★★'], [4, '★★★★']]

It can also be combined with string formatting:

>>> names = [('John', 'Matrix'), ('Jack', 'Slater')]
>>> names > foreach({'name': "{0} {1}", 'initials': '{0[0]}. {1[0]}.'}) | list
[{'initials': 'J. M.', 'name': 'John Matrix'},
 {'initials': 'J. S.', 'name': 'Jack Slater'}]

Automatic regex conditions

If you use a string instead of a function as a condition in where(), where_not(), select_first() or take_until() the string will be used as a regex to match the input against. This will, of course, work only if the items of the input sequence are strings.

Essentially:

where(r'^some\-regexp?$')

is equivalent to:

where(re.match, r'^some\-regexp?$')

As of 0.3.2 this also filters out None values instead of throwing an exception. Making it equivalent to:

where(maybe | (re.match, r'^some\-regexp?$'))

If you want to easily add this functionality to your own functions, you can use the regex_condition() decorator.