Ndtyjky

I have written some scrapy code to obtain HTML links from Indeed search page results. My start URL is a http address that provides a list of job ads. I am trying to scrape the URL for each job shown on the page and the job title. My problem appears to be the titles = response.xpath attribute. If I use a job specific attribute, I get data, but when I use the attribute shown below in my code I get nothing (not even the column headers). This is despite the fact that the attribute encompasses everything that I need. Any help welcomed, as I am just a beginner.

I'm outputting to a CSV file and I've used this code successfully elsewhere, so I'm wondering if it is something about the way they have coded the target URL page. It's driving me nuts!

from scrapy.spiders import Spider

from scrapy.selector import Selector

from ICcom4.items import Scrape4Item

from scrapy.linkextractors import LinkExtractor

from scrapy.utils.response import get_base_url

from scrapy.spiders import CSVFeedSpider

import requests



class MySpider(Spider):

    name = "Scrape4"

    allowed_domains = ["indeed.co.uk"]



    start_urls = ['http://www.indeed.co.uk/jobs?as_and=a&as_phr=&as_any=&as_not=IT+construction&as_ttl=Project+Manager&as_cmp=&jt=contract&st=&salary=%C2%A310K-%C2%A3999K&radius=25&l=&fromage=2&limit=50&sort=date&psf=advsrch',]



    def parse(self, response):

        titles = response.xpath('//div[@class="jobsearch-SerpJobCard row result clickcard"]')



        items = 

        for titles in titles:

            item = Scrape4Item()

            base_url = get_base_url(response)

            home_url = ("http://www.indeed.co.uk")

            item ['_pageURL'] = base_url

            item ['role_titletext'] = titles.xpath('//h2/a/text()').extract()



            items.append(item)

        return items

Thanks for the guidance Elena, but I'm afraid that your suggestions made no difference. I still get no data return. I have resolved the duplicate variable (for titles in titles1) which I tested as a standalone change satisfactorily. However, the other suggestions made no difference. I also tried running the scrape with just the request for a URL to be returned, and it still didn't work. Revised example is below.

from scrapy.spiders import Spider

from scrapy.selector import Selector

from ICcom4.items import Scrape4Item

from scrapy.linkextractors import LinkExtractor

from scrapy.utils.response import get_base_url

from scrapy.spiders import CSVFeedSpider

import requests



class MySpider(Spider):

    name = "Scrape4"

    allowed_domains = ["indeed.co.uk"]



    start_urls = ['http://www.indeed.co.uk/jobs?as_and=a&as_phr=&as_any=&as_not=IT+construction&as_ttl=Project+Manager&as_cmp=&jt=contract&st=&salary=%C2%A310K-%C2%A3999K&radius=25&l=&fromage=2&limit=50&sort=date&psf=advsrch',]



    def parse(self, response):

        titles1 = response.css('div.jobsearch-SerpJobCard.row.result.clickcard')

        #also tried as titles = response.css('div.jobsearch-SerpJobCard row result clickcard')



        items = 

        for titles in titles1:

            item = Scrape4Item()

            base_url = get_base_url(response)

            home_url = ("http://www.indeed.co.uk")

            item ['_pageURL'] = base_url

            item ['role_titletext'] = titles.xpath('.//h2/a/text()').extract()

        #also tried as item ['role_titletext'] = titles.css('h2 a::text').extract()

            items.append(item)

        return items

EDIT:
Thanks you Thiago. That's cracked it! You're a superstar!
Thanks to you and Elena for having patience with a newbie.
Just to complete the circle for anybody else, the final code that I used that worked was as below. This returns the search page url and the job title :-) ;

from scrapy.spiders import Spider

from scrapy.selector import Selector

from ICcom4.items import Scrape4Item

from scrapy.linkextractors import LinkExtractor

from scrapy.utils.response import get_base_url

from scrapy.spiders import CSVFeedSpider

import requests



class MySpider(Spider):

    name = "Scrape4"

    allowed_domains = ["indeed.co.uk"]

    start_urls = ['http://www.indeed.co.uk/jobs?as_and=a&as_phr=&as_any=&as_not=IT+construction&as_ttl=Project+Manager&as_cmp=&jt=contract&st=&salary=%C2%A310K-%C2%A3999K&radius=25&l=&fromage=2&limit=50&sort=date&psf=advsrch',]



    def parse(self, response):

        titles = response.css('.jobsearch-SerpJobCard')

        items = 

        for titles in titles:

            item = Scrape4Item()

            base_url = get_base_url(response)

            home_url = ("http://www.indeed.co.uk")

            item ['_pageURL'] = base_url

            item ['role_titletext'] = title.xpath('.//h2/a/@title').extract()

        items.append(item)

        return items

I noticed that there is no clickcard class int the downloaded HTML code, but it is there after page load. Surely it is added by some javascript code.
As Scrapy doesn't execute javascript, you may want to double check the page source when some selector fails unexpectedly (instead of 'inspect element').
Besides that, a shorter selector like '.jobsearch-SerpJobCard' would do the job.

Regarding the question in the title, to get an attribute data you may use xpath('.//div/@class') or css('div::attr(class)'). E.g:

def parse(self, response):

    titles = response.css('.jobsearch-SerpJobCard')

    for title in titles:

        item = {}

        item['role_titletext'] = title.xpath('.//h2/a/@title').get()

        # or

        # item['role_titletext'] = title.css('h2 a::attr(title)').get()

        yield item