No Results
Automation Example: Data Scraping
Article Contents

In this example, we want to scrape the number of GitHub repositories for the top programming languages.

For this purpose, we have created three visualizations and placed them on a page:

  • a table visualization created from an XLS feed containing a list of top programming languages we want to search on GitHub;
    You can download the spreadsheet here and then create an XLS feed.
  • Web Content connection to GitHub (endpoint: https://github.com) off of which we then created a feed, as seen in the screenshot.
    When you search a certain language on Git, the URL’s structure is https://github.com/search?q=grove, so when configuring our feed, in Start URl, we have entered /search?q= ; We have also inserted a node variable which will represent our search term (language). Our node variable is named gitSearch (but you can label it as you see fit) and is an unbounded string.
  • an HTML template containing a form that will be filled out with the name of the language that we have in the table and the value we get from Git — the number of repositories for the language in question.
    To configure the HTML template, you can use the following code in the HTML Renderer:

    <div class="container">
  <h1>
    {{title}}
  </h1>

  <form (ngSubmit)="onSubmit(formGroup)" novalidate>
    <div class="form-group">
      <label for="name">Name</label>
      <input type="text" class="form-control" id="name" name="name" />
    </div>

    <div class="form-group">
      <label for="name">Value</label>
      <input type="text" class="form-control" id="value" name="value" />
    </div>

    <button class="btn btn-primary" type="submit">Save</button>
  </form>
</div>

    And in the JavaScript Renderer:

    component.formGroup = {
      name: '',
      value: '',
};

component.onSubmit = (formData) => {
        var name = formData['name'];
       window.confirm("Successfully saved");
   };

This is the output:

We will create two scripts:
– The first script will search for a specific language on Git. Once that action is performed, Git will return the number of repositories the language is in.
– The second script will get the returned number of repositories and fill out the form with the name of the language and the number of repositories so we do not have to enter the information manually.

All the steps outlined below can be easily followed by watching the following video.

 

Step 1: Record the First Script

This script will search for a programming language on Git.

  1. On the page containing the visualizations, click Edit for the RPA Scripts button to appear.
  2. Click RPA Scripts.
    The minimized Script Center is displayed.
  3. Click Record New Steps.
    The recorder is displayed with available actions, and the record button is blinking.
  4. We want the language from the table to be filled out (copied) in Git, so we will click Ctrl + V in the Search GitHub field.
    The action is recorded and displayed in the recorder.
  5. Since we need to click on the Search button in order for Git to look for the search item and return results, we need to capture this click in the script. To do that, click Capture Next Mouse Click in HTML, and then click the Search button in Git.
    The action is recorded and displayed in the recorder.
  6. Click Stop Recording.
  7. Click Save Script As New.
    A pop-up is displayed.
  8. In the pop-up, in Script Name, enter the name for your script (for example, searchGit).
    Additionally, you can also provide a description.
  9. Save your changes.

 

Step 2: Create Run Script Action for the First Script

  1. In the visualization containing user information, click the three dots in the right corner and then click Edit Actions.
    A pop-up is displayed.
  2. In the pop-up, click + and then select Click Row > Run Script.
    A new window for configuring the action is displayed.
  3. In Base Config, do the following:
    a) In Action Name, provide a name for the action (for example search language).
    b) (Optional) Provide a description.
    c) (Optional) Enable or disable the Always Show in Menu toggle switch.
    When the switch is enabled, the action will appear when you click on a row in the table, and then you will need to click on it to perform it. When the switch is disabled, the action will not appear but will be performed when you click on a row.
  4. From the dropdown, select the script you have previously recorded.
  5. Click Next.
    You are taken to the Script Variables tab.
  6. In Script Variables, click Add Variable.
    For our example, we need to add one variable, as we have Languages.
    A pop-up is displayed.
  7. In the pop-up, in Variable Name, provide a name for the variable and click Save.
    We named the variable v1. The variable name is displayed, and now you need to set its mapping type and mapped value.
  8. In Mapping Type, select Record Value, and in Mapped Value, select a value. In our example, the mapped value for v1 is Languages.
  9. Click Next.
    You are taken to Conditions, where you can set the conditions under which the action will be performed. We will not set any conditions for our example. Click Next to go to Permissions. Here you can exclude certain roles from accessing the action. We will allow all users to access the action.
  10. Click Save and Close.
    The action is displayed in the pop-up. Close the pop-up.

 

Step 3: Modify the First Script

In the script that was created in step 1, we want to modify the first step to get its values from the variable in step 2 (v1).

  1. On the page containing the visualizations, click Edit for the RPA Scripts button to appear.
  2. Click RPA Scripts.
    The minimized Script Center is displayed.
  3. Click , load the script, and then click to maximize the Script Center.
    The 2 recorded steps are displayed: Set with JQuery Selector steps (the Ctrl + V command we used in step 1) and Click With JQuery Selector step (mouse click on the Search button).
  4. In the first step, click the field next to value.
    A pop-up for editing the value is displayed.
  5. In the pop-up:
    a) Click Advanced and select Set to Literal Value.
    b) Click again Advanced and then select Set to Variable Value.
    c) In Variable Name, enter the name of the variable. In our case, that is v1, as we created it in step 1 (its mapped value is Name).
    d) Click Save.
  6. (Optional) Set the delay in milliseconds between the steps by clicking the three dots in the top-right corner and selecting Add Delay. By default, the delay is 250 ms.
  7. Click to save your changes.
  8. Exit the Script Center.

This is what our first script looks like:

 

Step 4: Play the First Script

In this step, you will see your script in action.

  1. In the visualization/table containing languages, click on any row.
    The previously created Run Script action is displayed (if you enabled the Always Show in Menu toggle switch).
  2. Click on the action to run it.
    The name of the language is displayed in the Search GitHub field, and we see the search results, including the number of repository results.

 

Step 5: Record the Second Script

In Step 4, we got the number of repository results for a language we searched on Git, so now we will record a new script that will get the returned number of repository results and populate the HTML form with it. In our form, we have Name and Value fields, so this script will populate the Name field with the name of the language we searched on Git and Value with the number of repository results.

  1. Run the first script to get results like in step 4.
  2. Click the RPA Scripts button.
    The minimized Script Center is displayed.
  3. The first script is still loaded, so click Create a New Script.
  4. Click Record New Steps.
    The recorder is displayed with available actions, and the record button is blinking.
  5. We want the language from the table to be filled out (copied) in the form, so we will click Ctrl + V in the Name field.
    The action is recorded and displayed in the recorder.
  6. Since we want to get the number of repository results and populate the form without manually copying the numbers every time, in the recorder we will click the Get Text From Element button and then select the number of repository results. As seen in the screenshot above, we got 214,358 results for Swift, so we will select this text.
    The action is recorded and displayed in the recorder.
  7. We want to populate the Value field with the number of repository results we just got as text, so we will click Ctrl + V in the Value field.
    The action is recorded and displayed in the recorder.
  8. In our form, we have the Save button, so if we want to capture this click in the script we need to click Capture Next Mouse Click in HTML in the recorder, and then click the Save button in the form.
    The action is recorded and displayed in the recorder.
  9. Click Stop Recording.
  10. Click Save Script As New.
    A pop-up is displayed.
  11. In the pop-up, in Script Name, enter the name for your script (for example, getText).
    Additionally, you can also provide a description.
  12. Save your changes.

 

Step 6: Create Run Script Action for the Second Script

In this step, you will create a new action that will be used for the second script (the one recorded in step 5).

Create the action just like in step 2 above. All the steps are the same, but make sure you use a different name for this action so you can differentiate it from the one you created in step 2. Also, make sure you select the script that was created in step 5.

 

Step 7: Modify the Second Script

Just like in step 3, here you will modify the second script. The first step in the second script should also get its values from the variable in step 2 (v1).
You can follow all the steps outlined in step 3.

This should be the end result:

 

Step 8: Play Both Scripts

In this final step, you will play the two scripts you have previously created.

  1. In the visualization/table containing languages, click on any row.
    The two Run Script actions are displayed.
  2. Click on the first action you created (the one that will search Git) to run it.
    The name of the language is displayed in the Search GitHub field, and we see the search results, including the number of repository results.
  3. Click on the second action (the one that will get the number of repository results).
    The name of the language is displayed in the Name field, and the number of repository results is displayed in Value.
Terms | Privacy
×