Filtering

Filtering a table of a dm object may affect other tables that are connected to it directly or indirectly via foreign key relations.

dm_filter() can be used to define filter conditions for tables using syntax that is similar to dplyr::filter(). These conditions will be stored in the dm, and executed immediately for the tables that they are referring to.

With dm_apply_filters(), all tables will be updated according to the filter conditions and the foreign key relations.

dm_apply_filters_to_tbl() retrieves one specific table of the dm that is updated according to the filter conditions and the foreign key relations.

dm_filter(dm, table, ...)

dm_apply_filters(dm)

dm_apply_filters_to_tbl(dm, table)

Arguments

dm	A dm object.
table	A table in the dm.
...	Logical predicates defined in terms of the variables in .data, passed on to dplyr::filter(). Multiple conditions are combined with & or ,. Only the rows where the condition evaluates to TRUE are kept. The arguments in ... are automatically quoted and evaluated in the context of the data frame. They support unquoting and splicing. See vignette("programming", package = "dplyr") for an introduction to these concepts.

Value

For dm_filter(): an updated dm object (filter executed for given table, and condition stored).

For dm_apply_filters(): an updated dm object (filter effects evaluated for all tables).

For dm_apply_filters_to_tbl(), a table.

Details

The effect of the stored filter conditions on the tables related to the filtered ones is only evaluated in one of the following scenarios:

1. Calling dm_apply_filters() or compute() (method for dm objects) on a dm: each filtered table potentially reduces the rows of all other tables connected to it by foreign key relations (cascading effect), leaving only the rows with corresponding key values. Tables that are not connected to any table with an active filter are left unchanged. This results in a new dm class object without any filter conditions.

2. Calling dm_apply_filters_to_tbl(): the remaining rows of the requested table are calculated by performing a sequence of semi-joins (dplyr::semi_join()) starting from each table that has been filtered to the requested table (similar to 1. but only for one table).

Several functions of the dm package will throw an error if filter conditions exist when they are called.

Life cycle

These functions are marked "questioning" because it feels wrong to tightly couple filtering with the data model. On the one hand, an overview of active filters is useful when specifying the base data set for an analysis in terms of column selections and row filters. However, these filter condition should be only of informative nature and never affect the results of other operations. We are working on formalizing the semantics of the underlying operations in order to present them in a cleaner interface.

Use dm_zoom_to() and dplyr::filter() to filter rows without registering the filter.

Examples

dm_nyc <- dm_nycflights13()
dm_nyc_filtered <-
  dm_nycflights13() %>%
  dm_filter(airports, name == "John F Kennedy Intl")

dm_apply_filters_to_tbl(dm_nyc_filtered, flights)
#> # A tibble: 3,661 x 19
#>     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
#>    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
#>  1  2013     1    10        3           2359         4      426            437
#>  2  2013     1    10       16           2359        17      447            444
#>  3  2013     1    10      531            540        -9      832            850
#>  4  2013     1    10      535            540        -5     1015           1017
#>  5  2013     1    10      553            600        -7      711            715
#>  6  2013     1    10      557            600        -3      855            912
#>  7  2013     1    10      557            600        -3      933            925
#>  8  2013     1    10      558            600        -2      844            906
#>  9  2013     1    10      559            600        -1      653            658
#> 10  2013     1    10      607            601         6      932            918
#> # … with 3,651 more rows, and 11 more variables: arr_delay <dbl>,
#> #   carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
#> #   air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>

dm_nyc_filtered %>%
  dm_apply_filters()
#> ── Metadata ────────────────────────────────────────────────────────────────────
#> Tables: `airlines`, `airports`, `flights`, `planes`, `weather`
#> Columns: 53
#> Primary keys: 3
#> Foreign keys: 3

# If you want to keep only those rows in the parent tables
# whose primary key values appear as foreign key values in
# `flights`, you can set a `TRUE` filter in `flights`:
dm_nyc %>%
  dm_filter(flights, 1 == 1) %>%
  dm_apply_filters() %>%
  dm_nrow()
#> airlines airports  flights   planes  weather 
#>       16        3    11227     2345      861 
# note that in this example, the only affected table is
# `airports` because the departure airports in `flights` are
# only the three New York airports.

dm_nyc %>%
  dm_filter(planes, engine %in% c("Reciprocating", "4 Cycle")) %>%
  compute()
#> ── Metadata ────────────────────────────────────────────────────────────────────
#> Tables: `airlines`, `airports`, `flights`, `planes`, `weather`
#> Columns: 53
#> Primary keys: 3
#> Foreign keys: 3