Ndtyjky

Here's an example of the table of data I'm scraping:

The elements in red are in the <th> tags while the elements in green are in a <td> tag, the <tr> tag can be displayed according to how they're grouped (i.e. '1' is in it's own <tr>; HTML snippet:

EDIT: I forgot to add the surrounding div

<div class="table-cont">

    <table class="tg-1">

        <thead>

            <tr>

                <th class="tg-phtq">ID</td>

            </tr>

        </thead>

        <tbody>

            <tr>

                <td class="tg-0pky">1</td>

                <td class="tg-0pky">2</td>

                <td class="tg-0pky">3</td>

            </tr>

        </tbody>

    </table>

    <table class="tg-2">

        <thead>

            <tr>

                <th class="tg-phtq">Sample1</td>

                <th class="tg-phtq">Sample2</td>

                <...the rest of the table code matches the pattern...>

            </tr>

        </thead>

        <tbody>

            <tr>

                <td class="tg-0pky">Swimm</td>

                <td class="tg-dvpl">1:30</td>

                <...>

            </tr>

        </tbody>

        <...the rest of the table code...>

    </table>

</div>

As you can see, in the HTML they're actually two different tables while they're displayed in the above example as only one. I want to generate a JSON object where the keys and values include the data from the two tables as if they were one, and output a single JSON Object.

How I'm scraping it right now is a bit of modified javascript code I found on a tutorial:

EDIT: In the below, I've been trying to find a way to select all relevant <th> tags from both tables and insert them into the same array as the rest of the <th> tag array and do the same for <tr> in the table body; I'm fairly sure for the th I can just insert the element separately before the rest but only because there's a single one - I've been having problems figuring out how to do that for both arrays and make sure all the items in the two arrays map correctly to each other

EDIT 2: Possible solution? I tried using XPath Selectors and I can use them in devTools to select everything I want, but page.evaluate doesn't accept them and page.$x('XPath') returns JSHandle@node since I'm trying to make an array, but I don't know where to go from there

let scrapeMemberTable = async (page) => {

 await page.evaluate(() => {

  let ths = Array.from(document.querySelectorAll('div.table-cont > table.tg-2 > thead > tr > th'));

  let trs = Array.from(document.querySelectorAll('div.table-cont > table.tg-2 > tbody > tr'));

  // the above two lines of code are the main problem area- I haven't been 

  //able to select all the head/body elements I want in just those two lines of code

 // just removig the table id "tg-2" seems to deselect the whole thing

  const headers = ths.map(th => th.textContent);



  let results = ;



  trs.forEach(tr => {

      let r = {};            

      let tds = Array.from(tr.querySelectorAll('td')).map(td => td.textContent);



      headers.forEach((k,i) => r[k] = tds[i]);

      results.push(r);

  });



  return results; //results is OBJ in JSON format

 }

}



     ...



results = results.concat(            //merge into one array OBJ

        await scrapeMemberTable(page)

    );



     ...

Intended Result:

[

    {

        "ID": "1",                            <-- this is the goal

        "Sample1": "Swimm",

        "Sample2": "1:30",

        "Sample3": "2:05",

        "Sample4": "1:15",

        "Sample5": "1:41"

    }

]

Actual Result:

[

    {

        "Sample1": "Swimm",

        "Sample2": "1:30",

        "Sample3": "2:05",

        "Sample4": "1:15",

        "Sample5": "1:41"

    }

]