How to collect paginated API responses using spring boot WebClient?












2















I have a paginated response from an URL, I want to keep on hitting the next page URL which I get from the previous response and keep on collecting items till I don't have a "nextPage" URL in my response. How to achieve this in a reactive way using spring boot WebClient from WebFlux with out blocking?



Request1: 

GET /items
response:
{
items: [...]
nextPage: "/items?page=2"
}


Request2:

GET /items?page=2
response:
{
items: [...]
nextPage: "/items?page=3"
}


Request3:

GET /items?page=3
response:
{
items: [...]
nextPage: null
}


Here I have created mock urls
https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items
https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=2
https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=3



How can I extract all Items from the above responses in a reactive way without blocking?










share|improve this question





























    2















    I have a paginated response from an URL, I want to keep on hitting the next page URL which I get from the previous response and keep on collecting items till I don't have a "nextPage" URL in my response. How to achieve this in a reactive way using spring boot WebClient from WebFlux with out blocking?



    Request1: 

    GET /items
    response:
    {
    items: [...]
    nextPage: "/items?page=2"
    }


    Request2:

    GET /items?page=2
    response:
    {
    items: [...]
    nextPage: "/items?page=3"
    }


    Request3:

    GET /items?page=3
    response:
    {
    items: [...]
    nextPage: null
    }


    Here I have created mock urls
    https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items
    https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=2
    https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=3



    How can I extract all Items from the above responses in a reactive way without blocking?










    share|improve this question



























      2












      2








      2


      1






      I have a paginated response from an URL, I want to keep on hitting the next page URL which I get from the previous response and keep on collecting items till I don't have a "nextPage" URL in my response. How to achieve this in a reactive way using spring boot WebClient from WebFlux with out blocking?



      Request1: 

      GET /items
      response:
      {
      items: [...]
      nextPage: "/items?page=2"
      }


      Request2:

      GET /items?page=2
      response:
      {
      items: [...]
      nextPage: "/items?page=3"
      }


      Request3:

      GET /items?page=3
      response:
      {
      items: [...]
      nextPage: null
      }


      Here I have created mock urls
      https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items
      https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=2
      https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=3



      How can I extract all Items from the above responses in a reactive way without blocking?










      share|improve this question
















      I have a paginated response from an URL, I want to keep on hitting the next page URL which I get from the previous response and keep on collecting items till I don't have a "nextPage" URL in my response. How to achieve this in a reactive way using spring boot WebClient from WebFlux with out blocking?



      Request1: 

      GET /items
      response:
      {
      items: [...]
      nextPage: "/items?page=2"
      }


      Request2:

      GET /items?page=2
      response:
      {
      items: [...]
      nextPage: "/items?page=3"
      }


      Request3:

      GET /items?page=3
      response:
      {
      items: [...]
      nextPage: null
      }


      Here I have created mock urls
      https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items
      https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=2
      https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=3



      How can I extract all Items from the above responses in a reactive way without blocking?







      spring spring-boot reactive-programming spring-webflux project-reactor






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 16 '18 at 10:06







      karthikdivi

















      asked Nov 13 '18 at 5:45









      karthikdivikarthikdivi

      1,25911430




      1,25911430
























          2 Answers
          2






          active

          oldest

          votes


















          2














          Using expand, this can be achieved.
          Based on the mock urls provided by you.



          public Mono<List<Item>> getItems() {
          String url = "https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items";

          return fetchItems(url).expand(response -> {
          if (response.getNextPage() == null) {
          return Mono.empty();
          }
          return fetchItems(response.getNextPage());
          }).flatMap(response -> Flux.fromIterable(response.getItems())).collectList();
          }

          private Mono<Response> fetchItems(String url) {

          return client.get().uri(url).retrieve()
          .bodyToMono(Response.class);
          }





          share|improve this answer

































            1














            You can achive a desired effect using exapnd:



            @Test
            public void usingExpand(){

            Request innerData = new Request(null);
            Request middleData = new Request(innerData);
            Request rootData = new Request(middleData);

            Mono.just(rootData)
            .expand( t -> Mono.justOrEmpty(t.netxPage))
            .flatMap( t -> Flux.fromIterable(t.items))
            .subscribe(System.out::println);

            }

            public static class Request {
            List<String> items = new ArrayList<>();
            Request netxPage;

            public Request(Request netxPage) {
            this.items.add(UUID.randomUUID().toString());
            this.items.add(UUID.randomUUID().toString());
            this.netxPage = netxPage;
            }
            }


            The above code shuld produce the following result:



            dc78317c-5552-4723-90db-5392c67655be
            32ff12bb-5be1-415e-b481-dab85d9157dd
            cf1e3f36-a8e2-414d-90a2-7708eeedc5be
            91a6bc14-a396-483d-a66a-80bb98dc1968
            c95adae3-8e6f-489b-8a9d-4cea3080e150
            d6f8fe01-2c50-4574-958c-ec675331bb25


            Two UUID from each data object.






            share|improve this answer























              Your Answer






              StackExchange.ifUsing("editor", function () {
              StackExchange.using("externalEditor", function () {
              StackExchange.using("snippets", function () {
              StackExchange.snippets.init();
              });
              });
              }, "code-snippets");

              StackExchange.ready(function() {
              var channelOptions = {
              tags: "".split(" "),
              id: "1"
              };
              initTagRenderer("".split(" "), "".split(" "), channelOptions);

              StackExchange.using("externalEditor", function() {
              // Have to fire editor after snippets, if snippets enabled
              if (StackExchange.settings.snippets.snippetsEnabled) {
              StackExchange.using("snippets", function() {
              createEditor();
              });
              }
              else {
              createEditor();
              }
              });

              function createEditor() {
              StackExchange.prepareEditor({
              heartbeatType: 'answer',
              autoActivateHeartbeat: false,
              convertImagesToLinks: true,
              noModals: true,
              showLowRepImageUploadWarning: true,
              reputationToPostImages: 10,
              bindNavPrevention: true,
              postfix: "",
              imageUploader: {
              brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
              contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
              allowUrls: true
              },
              onDemand: true,
              discardSelector: ".discard-answer"
              ,immediatelyShowMarkdownHelp:true
              });


              }
              });














              draft saved

              draft discarded


















              StackExchange.ready(
              function () {
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53274568%2fhow-to-collect-paginated-api-responses-using-spring-boot-webclient%23new-answer', 'question_page');
              }
              );

              Post as a guest















              Required, but never shown

























              2 Answers
              2






              active

              oldest

              votes








              2 Answers
              2






              active

              oldest

              votes









              active

              oldest

              votes






              active

              oldest

              votes









              2














              Using expand, this can be achieved.
              Based on the mock urls provided by you.



              public Mono<List<Item>> getItems() {
              String url = "https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items";

              return fetchItems(url).expand(response -> {
              if (response.getNextPage() == null) {
              return Mono.empty();
              }
              return fetchItems(response.getNextPage());
              }).flatMap(response -> Flux.fromIterable(response.getItems())).collectList();
              }

              private Mono<Response> fetchItems(String url) {

              return client.get().uri(url).retrieve()
              .bodyToMono(Response.class);
              }





              share|improve this answer






























                2














                Using expand, this can be achieved.
                Based on the mock urls provided by you.



                public Mono<List<Item>> getItems() {
                String url = "https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items";

                return fetchItems(url).expand(response -> {
                if (response.getNextPage() == null) {
                return Mono.empty();
                }
                return fetchItems(response.getNextPage());
                }).flatMap(response -> Flux.fromIterable(response.getItems())).collectList();
                }

                private Mono<Response> fetchItems(String url) {

                return client.get().uri(url).retrieve()
                .bodyToMono(Response.class);
                }





                share|improve this answer




























                  2












                  2








                  2







                  Using expand, this can be achieved.
                  Based on the mock urls provided by you.



                  public Mono<List<Item>> getItems() {
                  String url = "https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items";

                  return fetchItems(url).expand(response -> {
                  if (response.getNextPage() == null) {
                  return Mono.empty();
                  }
                  return fetchItems(response.getNextPage());
                  }).flatMap(response -> Flux.fromIterable(response.getItems())).collectList();
                  }

                  private Mono<Response> fetchItems(String url) {

                  return client.get().uri(url).retrieve()
                  .bodyToMono(Response.class);
                  }





                  share|improve this answer















                  Using expand, this can be achieved.
                  Based on the mock urls provided by you.



                  public Mono<List<Item>> getItems() {
                  String url = "https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items";

                  return fetchItems(url).expand(response -> {
                  if (response.getNextPage() == null) {
                  return Mono.empty();
                  }
                  return fetchItems(response.getNextPage());
                  }).flatMap(response -> Flux.fromIterable(response.getItems())).collectList();
                  }

                  private Mono<Response> fetchItems(String url) {

                  return client.get().uri(url).retrieve()
                  .bodyToMono(Response.class);
                  }






                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited Nov 19 '18 at 9:39

























                  answered Nov 19 '18 at 7:57









                  AshishAshish

                  564




                  564

























                      1














                      You can achive a desired effect using exapnd:



                      @Test
                      public void usingExpand(){

                      Request innerData = new Request(null);
                      Request middleData = new Request(innerData);
                      Request rootData = new Request(middleData);

                      Mono.just(rootData)
                      .expand( t -> Mono.justOrEmpty(t.netxPage))
                      .flatMap( t -> Flux.fromIterable(t.items))
                      .subscribe(System.out::println);

                      }

                      public static class Request {
                      List<String> items = new ArrayList<>();
                      Request netxPage;

                      public Request(Request netxPage) {
                      this.items.add(UUID.randomUUID().toString());
                      this.items.add(UUID.randomUUID().toString());
                      this.netxPage = netxPage;
                      }
                      }


                      The above code shuld produce the following result:



                      dc78317c-5552-4723-90db-5392c67655be
                      32ff12bb-5be1-415e-b481-dab85d9157dd
                      cf1e3f36-a8e2-414d-90a2-7708eeedc5be
                      91a6bc14-a396-483d-a66a-80bb98dc1968
                      c95adae3-8e6f-489b-8a9d-4cea3080e150
                      d6f8fe01-2c50-4574-958c-ec675331bb25


                      Two UUID from each data object.






                      share|improve this answer




























                        1














                        You can achive a desired effect using exapnd:



                        @Test
                        public void usingExpand(){

                        Request innerData = new Request(null);
                        Request middleData = new Request(innerData);
                        Request rootData = new Request(middleData);

                        Mono.just(rootData)
                        .expand( t -> Mono.justOrEmpty(t.netxPage))
                        .flatMap( t -> Flux.fromIterable(t.items))
                        .subscribe(System.out::println);

                        }

                        public static class Request {
                        List<String> items = new ArrayList<>();
                        Request netxPage;

                        public Request(Request netxPage) {
                        this.items.add(UUID.randomUUID().toString());
                        this.items.add(UUID.randomUUID().toString());
                        this.netxPage = netxPage;
                        }
                        }


                        The above code shuld produce the following result:



                        dc78317c-5552-4723-90db-5392c67655be
                        32ff12bb-5be1-415e-b481-dab85d9157dd
                        cf1e3f36-a8e2-414d-90a2-7708eeedc5be
                        91a6bc14-a396-483d-a66a-80bb98dc1968
                        c95adae3-8e6f-489b-8a9d-4cea3080e150
                        d6f8fe01-2c50-4574-958c-ec675331bb25


                        Two UUID from each data object.






                        share|improve this answer


























                          1












                          1








                          1







                          You can achive a desired effect using exapnd:



                          @Test
                          public void usingExpand(){

                          Request innerData = new Request(null);
                          Request middleData = new Request(innerData);
                          Request rootData = new Request(middleData);

                          Mono.just(rootData)
                          .expand( t -> Mono.justOrEmpty(t.netxPage))
                          .flatMap( t -> Flux.fromIterable(t.items))
                          .subscribe(System.out::println);

                          }

                          public static class Request {
                          List<String> items = new ArrayList<>();
                          Request netxPage;

                          public Request(Request netxPage) {
                          this.items.add(UUID.randomUUID().toString());
                          this.items.add(UUID.randomUUID().toString());
                          this.netxPage = netxPage;
                          }
                          }


                          The above code shuld produce the following result:



                          dc78317c-5552-4723-90db-5392c67655be
                          32ff12bb-5be1-415e-b481-dab85d9157dd
                          cf1e3f36-a8e2-414d-90a2-7708eeedc5be
                          91a6bc14-a396-483d-a66a-80bb98dc1968
                          c95adae3-8e6f-489b-8a9d-4cea3080e150
                          d6f8fe01-2c50-4574-958c-ec675331bb25


                          Two UUID from each data object.






                          share|improve this answer













                          You can achive a desired effect using exapnd:



                          @Test
                          public void usingExpand(){

                          Request innerData = new Request(null);
                          Request middleData = new Request(innerData);
                          Request rootData = new Request(middleData);

                          Mono.just(rootData)
                          .expand( t -> Mono.justOrEmpty(t.netxPage))
                          .flatMap( t -> Flux.fromIterable(t.items))
                          .subscribe(System.out::println);

                          }

                          public static class Request {
                          List<String> items = new ArrayList<>();
                          Request netxPage;

                          public Request(Request netxPage) {
                          this.items.add(UUID.randomUUID().toString());
                          this.items.add(UUID.randomUUID().toString());
                          this.netxPage = netxPage;
                          }
                          }


                          The above code shuld produce the following result:



                          dc78317c-5552-4723-90db-5392c67655be
                          32ff12bb-5be1-415e-b481-dab85d9157dd
                          cf1e3f36-a8e2-414d-90a2-7708eeedc5be
                          91a6bc14-a396-483d-a66a-80bb98dc1968
                          c95adae3-8e6f-489b-8a9d-4cea3080e150
                          d6f8fe01-2c50-4574-958c-ec675331bb25


                          Two UUID from each data object.







                          share|improve this answer












                          share|improve this answer



                          share|improve this answer










                          answered Nov 14 '18 at 21:08









                          piotr szybickipiotr szybicki

                          584210




                          584210






























                              draft saved

                              draft discarded




















































                              Thanks for contributing an answer to Stack Overflow!


                              • Please be sure to answer the question. Provide details and share your research!

                              But avoid



                              • Asking for help, clarification, or responding to other answers.

                              • Making statements based on opinion; back them up with references or personal experience.


                              To learn more, see our tips on writing great answers.




                              draft saved


                              draft discarded














                              StackExchange.ready(
                              function () {
                              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53274568%2fhow-to-collect-paginated-api-responses-using-spring-boot-webclient%23new-answer', 'question_page');
                              }
                              );

                              Post as a guest















                              Required, but never shown





















































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown

































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown







                              Popular posts from this blog

                              Florida Star v. B. J. F.

                              Danny Elfman

                              Lugert, Oklahoma