Quartz 4

LinkedIn-Leads kostenlos scrapen mit n8n & Google Custom Search API

17 min read

LinkedIn-Leads kostenlos scrapen mit n8n & Google Custom Search API

Haupterkenntnis: Mit der kostenlosen Google Custom Search API und n8n lassen sich täglich bis zu 1.000 LinkedIn-Profile vollautomatisch und ohne Kosten scrapen.

Creator: Bart Slodyczka
Branche: Marketing
Bereich: Automation

Tags: linkedin, lead-generation, n8n, google-custom-search-api, scraping, google-sheets, no-code, pagination

Kurzbeschreibung

Schritt-für-Schritt-Tutorial, das zeigt, wie man mit n8n und der kostenlosen Google Custom Search API LinkedIn-Profile automatisiert scrapt und in Google Sheets speichert – ohne bezahlte APIs.

Langbeschreibung

Der Creator Bart Slodyczka zeigt, wie man einen n8n-Workflow baut, der über die Google Custom Search API gezielt LinkedIn-Profile durchsucht (z. B. „CEOs in der Automobilindustrie, USA”), die Ergebnisse per Code-Node extrahiert (Name, Titel, Profil-URL, Snippet) und automatisch in Google Sheets schreibt. Durch Pagination werden bis zu 10 API-Calls pro Suchanfrage geloopt, was 100 Profile pro Query ergibt. Im kostenlosen Tier sind 100 API-Anfragen pro Tag möglich – das entspricht 1.000 LinkedIn-Leads täglich bzw. 7.000 pro Woche, vollständig gratis.

Stichpunkte

  • Google Custom Search API als kostenloser LinkedIn-Scraper-Ersatz
  • Free Tier: 100 Requests/Tag × 10 Ergebnisse = 1.000 Leads täglich
  • n8n-Workflow mit HTTP Request Node, Code Node, Wait Node, Google Sheets Node
  • Pagination-Loop bis max. 10 Seiten (konfigurierbar via maxPages-Variable)
  • Start Index beginnt bei 1, erhöht sich jeweils um 10 pro Iteration
  • 5-Sekunden-Wait verhindert Rate-Limit-Fehler
  • Ausgabe: Name, Titel, LinkedIn-URL, Snippet → direkt in Google Sheets
  • Setup benötigt: Google Cloud Projekt, Custom Search API aktivieren, API Key + CX-Wert
  • Methode übertragbar auf andere Domains (nicht nur LinkedIn)

Zitate

“You can do 7,000 new leads fresh from LinkedIn for free per week, which I think is absolutely fantastic for this method.”

Action Items

  1. Google Cloud Console öffnen → neues Projekt erstellen
  2. “Custom Search API” suchen und aktivieren
  3. Credentials → API Key erstellen und kopieren
  4. Programmable Search Engine erstellen (programmablesearchengine.google.com) → CX-Wert kopieren
  5. In n8n: HTTP Request Node anlegen, GET-Request an https://www.googleapis.com/customsearch/v1, Parameter setzen: key (API Key), cx (CX-Wert), q (Suchstring z. B. site:linkedin.com/in CEO automotive industry United States), start (dynamisch per Expression)
  6. Set Fields Node: currentStartIndex = 1, maxPages = 10
  7. Code Node nach HTTP Request: extrahiert Name, Titel, URL, Snippet aus API-Antwort
  8. Wait Node: 5 Sekunden Pause einbauen
  9. Google Sheets Node: extrahierte Felder in Spalten mappen
  10. Pagination Node: continueLoop und neuen startIndex prüfen → bei true zurück zum HTTP Node loopen
  11. Suchanfrage nach Bedarf anpassen (andere Branchen, Titel, Länder)

Full Transcript

Hello legends. In this video, I’m going to show you how to scrape LinkedIn profiles to generate new leads without paying for any API’s whatsoever, so that you can create spreadsheets of LinkedIn leads like this that have the leads name, the title, the link to their profile and a snippet of information from their profile. And each of these links actually goes to that person’s profile that you can then use to either add them or DM them or to run some other automation to find information about the company. And before I actually get into the automation, I just want to show you the method that we’re using here is actually we’re basically doing a Google search where we’re searching the LinkedIn site specifically for people that fit this profile, so we’re looking for CEOs in the automotive industry in United States. And if I just copy this, I want to show you what we’re actually doing here. So I’m just pasting that same search string directly into Google, and if I hit enter, you can see here that all of these results are the same results that we’re pulling out from our NAN scenario. So the first one was Jim Farley, which is the CEO of Ford Motor Company, and then we’ve got him over here, Jim Farley CEO of Ford Motor Company. So really what we’re doing is we’re actually using the Google search API to just search for people on LinkedIn, and then we’re returning all these results, we’re scraping them and adding them to our Google sheets. All right, so let’s demonstrate how this works. I’m just going to delete all of these search results so you can see that we’re doing that we’re starting from a first string. And back in NAN, I’m going to open up my chat. I’m just going to say hi, and now we’re triggering off, and we’re actually executing our sequence. So we’re hitting that search API to receive all the results. We’re extracting all the results. We’re waiting for a little bit so we actually don’t hit any API limits for the Google search API calls that we’re doing or for the add to the Google sheets API calls. I’m just going to stop this here because the list is going to be built anyway. That’s what the wait block is for. So yeah, after we hit the HTTP request, we get all the results, which are actually 10 per request, then we’re adding them to the Google sheets. And then we’re just checking, are there any more results from that same query that we initially hit? And if there are more results, we’re basically looping back into this module over here, and we’re just going to page two. Now every API that you hit for the Google search API, you only get 10 results back. So technically, if you want to get more and more results, you have to hit that API again. All right, guys, just quickly, I have a wait list for an AI community that I’m going to be building at a high level. These are the things that I’m going to be offering. So like weekly calls behind the scenes of my NAN make.com builds, any custom coded stuff that I do, any other interesting things that I work on. I’m actually running this community with my brother, Chris, who has run a lead generation, like a lead gen agency. He also set up Jewelker.com’s full sales and marketing pipelines in the team. So he’s very experienced with sales and marketing. I’m very good with AI and automation. So together, I think we have a blend, like a unique blend of an offering. And then the starting price is going to be at 55 bucks. So if you feel like the questions below, it’s basically your, your name, your email, and then some information about like how long have you been following my channel? And then enjoy the most. And then what’s a must have in a community that will make it worth the money for you. Thank you very much guys. Let’s get back into the video. But now over in our Google sheets, you can see that we got the same results as before. So we had the first 10 from before. But now we did the pagination, which is we’re basically going to page two and getting another 10 results and another 10 results and another 10 results. So I stopped ours prematurely. But as you can see here, we’re able to generate a list of leads extremely quickly. And then we have links to all the leads that we want to contact. Alright, so let’s break down each module so we can understand how this works. Let’s dig into the set fields. So in the set fields, we’re actually setting two variables. One is called current star index. We’re setting that to one. And then the other one is called max pages and we’re setting it to 10. Now each API call to the Google search API brings us back 10 search results or 10 LinkedIn profiles. I’m just setting us a limit. So we have up to 10 API calls to that one search string that we’re doing. So we can get 100 leads that are CEOs in the automotive industry in the United States. But you can set this higher if you do one. And then you can just generate even more leads for that specific query. So zooming out the very next thing we’re doing is we’re actually hitting that search API. So this is where we actually have to create some credentials so we can actually hit the search API. But at a high level, we’re just making a get request to the Google APIs forward slash custom search endpoint, which basically allows us to make a Google search using a specific search string. And then we’re setting a bunch of query parameters. So these query parameters are actually appended into this URL, what to make it easy for you guys to duplicate this flow. I’ve just broken them down like this. So we have the API key so we can actually hit the API. Then we have the CX value from another setting we have to create. And I’ve got both of those links at the start of this workflow. So just give me a little bit and I’ll get to there in a second. And then we have the actual search string. So Q is the name for the search string. And we’re basically searching for CEOs in the automotive industry in the United States. And then at the very bottom, we have our start value. So the start value represents which page do we want to start on when we’re hitting the API call for this search request because technically we could just start at page 10 and then we’re good to go. But what we really want to do is we want this to be dynamic for the very first request. We want to start on page one. But then every time we make an iteration through we actually want to pull out that we want to go to page two, page three, page four. So the rest of the code nodes in here actually do that for us automatically. But we have to run this simple expression. So we’re asking ourselves is the run index equal to zero, which is is this the very first time we’re running this API call. And if it is, then we’re basically taking that value from the set fields block. So the set fields block, we’re taking that current start index value, which is set to one and then we’re making this API call. But as soon as we go one rotation around and now we’re on run to this statement is no longer true. We’re not actually on the first run run the second run. So we now go into this expression and this expression is the very last module that we have in this workflow and it basically checks, oh, is there another page of results. If there is, let’s loop back around until we hit 10 full pages. Sometimes you might have a very specific, very niche search that you’re looking for and it might only have three search like three pages of results, not 10. Like this is probably a very common string, but you might have whatever you’re looking for might be very, very specific and you might only have three pages. So you don’t be wasting the, you know, fourth, fifth, sixth API calls for no reason. This flow will actually just stop you. But yeah, this is what this equation is doing. So then for the second, third, fourth API call that we’re running through, we’re just using the page two, page three, page four. And that’s what we’re setting from the actual node later on. So for the very first one, we’re actually using the set fields, which is set to one. For this instance, we’ve actually fired up a few times the actual value is 31 in this case. And not to show you the output of the API call on the right hand side, you can see if I zoom into here, we have previous page, we had count 10, which is 10 results that we had in a start index was 21. So essentially going, when it, when this value equals to one, we’re actually looking for the very first search result. So we’re starting on page one for the very first search result. And then we’re plusing 10. So that means when we go to here, when this equals 11, it’s actually the start of page two, when it equals 21, it’s a start of page three, and then so on and so forth. So we’re actually plusing an extra 10 each time, simply because there are 10 results per API call. But anyway, I’m just going to zoom out of here, and then we have all the API results. So we have pure, which is the president CEO of Bendix commercial vehicle, then we have all the information information associated with him, and it can kind of keep scrolling down and then you have the rest of the 10 results. Now, to set up your credentials, I’ve got some information over here. So there’s actually two things that you do. The first is you need to go to your Google Cloud console, and then you want to create yourself a new project. So I’m just going to click on my project, then I’m going to go to new project. I’m going to call this linked in dash two. So I really got linked in one and Google is a little bit tricky. So even though we created the project, it’s not going to go into that project for us straight away. So we have to just select our project. Once you’ve selected your project, let’s go back to NAN. We want to search for and enable the custom search API. So let’s just type that in custom search API. Let’s click on it. Awesome. Let’s go enable. All right. Awesome. It’s turned on. And now we want to go to credentials and then create an API key and then copy our API key value. So I’m going to go credentials. I’m going to go create credentials API keys. I’m going to copy this value. I’m going to close this and now I’m linked in scraper. I want to go back into our HTTP module and over here where it has key and then the value. I’m just going to backspace this. Paste this one in and now I’m just going to leave that to save. And the second thing we need to do is create a programmable search engine. So let’s click on this link. So our instructions are create a name, leave it as search entire web and then copy the CX value from the field that is created. Okay. So I’m going to go Google dash search. And then I’m going to go search entire web. You can technically just put the LinkedIn domain over here. But if you do want to use this method for other websites, then, you know, just leave it search the entire web. So I want to go on. I’m not a robot. Click create, which is zoom out. And once this is ready, we have our CX value, which is this little bit over here at the end of this URL. So I’m going to copy that control C. Go back into LinkedIn, double click on this and what has CX value on a backspace and then paste this in. And now one more thing I want to speak about quickly. So what we’re doing is we’re using the free tier for the Google search API, which gives us 100 requests per day. Now each of those requests returns 10 search results, which means that 100 requests per day by 10 search results is 1000 lead will essentially 1000 search results or 1000 LinkedIn leads per day that you can run at no cost. Now you can upgrade your tier and just pay a little bit more. And then you can do essentially unlimited, like whatever you want to pay for, you can just amp it up. But this means that you can do 7,000 new leads fresh from LinkedIn for free per week, which I think is absolutely fantastic for this method. So that’s one thing to keep in mind. We’ll come back into here and actually test it out, but let’s go into the extract values now. Okay, so what’s happening in this code node, you can go through this code, you can put it into chat GPT to explain it for you, but essentially we’re going through this API response, which again is like full of some stuff that we probably don’t need. And it’s pulling out each of those 10 LinkedIn profiles and then pulling out their name, the snippet, the actual URL to their LinkedIn profile, and that information that we’re essentially adding to Google Sheets. And then it’s taking this, it’s cleaning it up with the code, and then it’s putting the search results here. So you can see we have pilot over here, we have the title, the link to snippet, the image, if I just zoom in, and then we have the next person is the next result Edward, we have his title link snippet image, everything else. And this is just for our sake when we’re doing our page nation checks. So we’re seeing what start index are we at, and is there any more results in the very next page. Then like I described before, we essentially have this function, which is just waiting for five seconds. That’s because we don’t want to hit any API limits for the HTTP code that we’re making for the Google search API, or when we’re adding information into the Google Sheets as well. So this is five seconds, I find that it works well for me, but if you run into any issues, you maybe just need to pump it up by another five seconds to 10. Then for our Google Sheets module, we’re essentially just taking all the information bit by bit from the previous node, which is the code block, and then we’re adding it, and we’re mapping it into the, yeah, into the Google Sheets value by value. All right, so let’s go into the next block, then we have a page, a page nation over here. So this is pretty simple. We’re just going through the results over here. We’re just double checking that, OK, firstly, what index are we at now, so we know what index we need to be at for the next HTTP call to the Google search API. And we’re just confirming, all right, are there any other search results after this page. So in our case, we had these search results after we run it through the code, then we had this information brought back to us. So we had continued loop equals true, which means, yep, let’s keep going right around, and then we had start index equals 31, which means that, all right, now we need to plug 31 into the HTTP call, so we know we need to start on the very next page and get that very first result. So that’s why we have over here, if that condition is true, if we actually do have more results, we go back into this HTTP call, and then if I scroll down over here, and I zoom into the start value, you can see that we have 31, just like what we had in that previous check node as well. So we had 31 here because it was 31 there. OK, also, so let me just delete the results that we had over here, and let’s just confirm that the Google authentication method that we ran through just before is working. So this is a blank sheet. Let’s go to refresh this reset, I’m just going to say, hi, hit enter, and now we’re shooting off and then getting our 10 results each time we’re waiting, let’s just confirm that they’re added to the sheet appropriately. So let me just pause this now, we should have our first 10 results, and then we go our first 10 results are here. Awesome. All right, guys, thank you very much for watching. I hope you enjoyed this video. If you did enjoy the video, can you please drop a comment below, like the video as well. It’s going to help the algorithm and it’s going to help my channel grow. Thank you. See you.

Graph View

Backlinks