Filtering and Paginating Responses

How to filter and paginate responses from the LMS API.

Introduction

The LMS API will often respond with objects that contain a list of items rather than just a single item. This is most common with the following:

  • GET requests to OData endpoints where an ID is not specified 
  • POST requests to Elasticsearch search endpoints 

Response Objects with Multiple Results

By default, OData will return 50 results and Elasticsearch will return 10. These defaults are in place to limit endpoint bandwidth and lower response times while still providing an abundant amount of information.

However, removing these limits may be necessary in certain circumstances. On this page, we'll describe how to use endpoint parameters so that you can filter the information you receive in your requests.

Let's begin by looking at an example OData endpoint:

https://loanpro.simnang.com/api/public/api/1/odata.svc/Payments?$filter=entityId eq 5919&$top=30

This is an example of the Payments endpoint with a few filtering parameters.

LMS endpoint parameters are tacked on to the endpoint URL by first adding a single question mark (?) at the end of the URL before the parameter list. Then, the parameter name is added and is preceded by a dollar sign ($). The parameter name is followed by an equals sign (=) and a value. Lastly, if you are using multiple endpoint parameters, an ampersand (&) is placed between each one. Listed below is the order of a parameter list as well as examples of endpoints with parameters.

Here's an example OData endpoint:

Example Description
https://loanpro.simnang.com/api/public/api/1/ The base URL for all LMS endpoints.
odata.svc/Payments The OData endpoint for Payments.
? HTTP syntax. The question mark indicates that the regular endpoint has ended, and everything that follows is an endpoint parameter that will limit or expand the results.
$filter The parameter name. The dollar sign indicates that it is an OData token that can carry out a specific function beyond a normal parameter that simply holds a value.
=entityId eq 5919 The value. The equals sign connects this to the parameter name that preceded it.
& HTTP syntax. The ampersand distinguishes between parameters.
$top A second parameter name, indicating how many results should be shown.
=30 The value matching the $top parameter name. It tells the system we want the top 30 entries.

And here's an example Elasticsearch endpoint:

Example Description
https://loanpro.simnang.com/api/public/api/1/ The base URL for all LMS endpoints.
Loans/Autopal.Search() The Elasticsearch endpoint for searching loans.
? HTTP syntax. The question mark indicates that the regular endpoint has ended, and everything that follows is an endpoint parameter that will limit or expand the results.
$start The parameter name, indicating where the results should begin.
=20 The value paired to the preceding parameter. In this case, it tells the system to start the results at the twentieth entry.
& HTTP syntax. The ampersand distinguishes between parameters.
$top A second parameter name, indicating how many entries should be shown.
=30 The value matching the $top parameter name. It tells the system we want the top 30 entries.

Currently, the LoanPro LMS OData implementation only supports paging parameters at the root level and not for nested associated entities. Due to this, you'll need to submit requests to the correct endpoints.
A desired request such as
GET /odata.svc/Loans(5919)/Payments?$top=30
will not work as assumed, and should instead be written as
GET /odata.svc/Payments?$filter=entityId eq 5919&$top=30

Paging with OData Requests

Set Page Size

Setting a page size allows you to determine a maximum number of results you'd like to receive. To do so, you can use the $top parameter within your requests. This does not guarantee that the specified amount of results will be returned, and it will not change the order the items are returned. But it will guarantee that no more than the specified number of items are returned.

https://loanpro.simnang.com/api/public/api/1/odata.svc/Payments?$filter=entityId eq 5919&$top=200

Skip and Skiptoken

We've covered altering the page size, but what` about retrieving results from a specified point and beyond? To achieve this, use the $skip and $skiptoken parameters. The $skip parameter allows you to do exactly that: skip however many results you'd like and return the results that follow.

https://loanpro.simnang.com/api/public/api/1/odata.svc/Payments?$filter=entityId eq 5919&$skip=50

Where $skip specifies a static number of results to skip, $skiptoken allows you to specify an exact ID from the database for the entity to skip past and return the results that follow.

https://loanpro.simnang.com/api/public/api/1/odata.svc/Payments?$filter=entityId eq 5919&$skiptoken=239898

Both $skip and $skiptoken are non-inclusive, meaning wherever you skip to will not be included in the results (e.g. $skip=50 will not include result 50; returned results will start at 51)

Filter

The $filter parameter allows you to restrict the result set that will be returned without having to use an Elasticsearch query object.

This parameter can be used on any property returned in a list item for the endpoint and follows this basic structure:

Here's a breakdown of each section of the endpoint:

Example Description
https://loanpro.simnang.com/api/public/api/1/ The base URL for all LMS endpoints.
odata.svc/Payments The OData endpoint for Payments.
? HTTP syntax. The question mark indicates that the regular endpoint has ended, and everything that follows is an endpoint parameter that will limit or expand the results.
$filter The parameter name, indicating the results limitation.
=entityId eq 5919 The value paired to the preceding parameter. In this case, it tells the system to only retrieve results who have an entityId equal to 5919.
and Keyword to separate expressions in the same $filter parameter.
date lt datetime'2021-10-04T00:00:00' The second value paired to the preceding parameter. In this case, it tells the system to only retrieve results who have a date that is less than the date of 10/04/2021 at 00:00:00.

The $filter parameter supports the following comparison operators: eq (equal), lt (less than), gt (greater than), le, (less than or equal to), and ge (greater than or equal to).

Additionally, any string values being compared to should be contained in single quotes ('string'). This includes any dates, datetimes, words, sentences, etc.

You can use $filter with any name:value pair from the JSON object that's returned by default in the response from the entity. For example, the response from a payment list item includes a comment field. You could add a filter token to find only the payments with a specific comment: $filter=comments eq 'A Specific Comment' 

The LMS API supports a few other OData filter parameters. Here's a list of other available parameters you can use:

Category Available Filter Options Syntax
Logical Operators equals, not equals eq, ne
  greater than, greater than or equal to gt, ge
  less than, less than or equal to lt, le
  and, or and, or
  has has
  in in
Arithmetic Operators and Functions addition, subtraction add, sub
  multiplication, division mul, div
  modulo mod
  grouping  
  floors, ceilings floor, ceiling
  round round
String and Collection concatenation concat
  string ends with, string starts with endswith, startswith
  whitespace trimming trim
  to lower, to upper tolower, toupper

Select

Finally, the $select parameter allows you to list specific information for each filter result. For example, if you filter Payment information (like shown above) and would like to list the payment amount for every result, add $select=amount to your request endpoint. This is a useful tool for selecting only necessary information wihtin your responses. You can also select multiple parameters by separating each one with a comma; for example, you can use $select=id,amount,date to see the ID of a payment, the amount that was paid, and on which day.

Here's an example of the entire endpoint:

https://loanpro.simnang.com/api/public/api/1/odata.svc/Payments?$filter=entityId eq 9100&$top=200&$select=id,amount,date

Response Summary

Each OData request response will contain a small summary object with the following properties:

  • start - The index at which the results list starts
  • pageSize - The set page size
  • total - The number of results returned in the results list

Paging with Elasticsearch Requests

Set Page Size

Setting page size in the Elasticsearch endpoints works in the same way as the OData endpoints. If you'd like to set the page size, you can do so with the $top parameter. Again, this does not guarantee that the specified amount of results will be returned, and it will not change the order the items are returned, but it will guarantee that no more than the specified number of items are returned.

https://loanpro.simnang.com/api/public/api/1/Loans/Autopal.Search()?$top=50

Start

The OData parameters of $skip and $skiptoken are not supported for the search endpoints. Instead, these requests utilize the $start parameter to specify a starting place for the results. For example, using $start for search endpoints functions the same way as $skip for OData.

https://loanpro.simnang.com/api/public/api/1/Loans/Autopal.Search()?$start=50

Response Summary

Each search request response will contain a small summary object with the following properties:

  • totalHits - The total number of results returned from the query (this will not match the results contained in the results list, as this is handled with paging)
  • totalTime - The amount of time it took to retrieve results

Elasticsearch Safeguard

Elasticsearch has a built-in safeguard that prevents you from paging through more than 10,000 results. This means that if the sum of your $start and $top parameters is greater than 10,000, Elasticsearch will fail to return any results.

However, you can circumvent this issue by fetching results beyond the first 10,000 with the search_after payload parameter.