I’ve been busy refreshing my Python skills by working through the excellent udemy, 100 Days of Code: The Complete Python Pro Bootcamp for 2022 course by Dr. Angela Yu.
I’m currently working on day 48 which involves scrapping the Upcoming Events section of the Python.org website using the Selenium WebDriver. In the example code, Dr. Yu shows how to write the output of the scrapping operation into a dictionary using a for loop but says that it also could be accomplished using dictionary comprehension. I decided that would be an interesting challenge and would help to reinforce my knowledge of dictionary comprehension so I gave it a try.
I thought that it would be a fairly easy thing to do but it turned out to be more complex that I anticipated. I did a search for examples and came up with nothing so I’ve decided to share what I found here, hoping that it might help someone else.
Lets start by showing the desired output structure:
and here’s the code that I used to accomplish the task:
from selenium import webdriver
from selenium.webdriver.common.by import By
chrome_driver_path = "C:/Development/chromedriver.exe"
THE_URL = "https://www.python.org/"
driver = webdriver.Chrome(executable_path=chrome_driver_path)
driver.get(THE_URL)
event_times = driver.find_elements(By.CSS_SELECTOR, value=".event-widget time")
event_names = driver.find_elements(By.CSS_SELECTOR, value=".event-widget li a")
events = {}
events = {x: {"time": value[0].text, "name": value[1].text} for x, value in enumerate(zip(event_times, event_names))}
print(events)
driver.quit()Most of it is pretty straight forward. I use the find_elements function to create two lists containing times and event using the CSS_SELECTOR as the key. Next comes the fun part, using dictionary comprehension to massage the results into the desired format.
Here’s an explanation of what’s going on:
The zip function combines the two list together allowing you to access both the time and name data simultaneously during each iteration of the loop. If we were just creating a dictionary with time and event data this would be alright but we are really trying to create a set of dictionaries within dictionaries with the key to outer dictionary being an integer value from 0..n. To accomplish this, we need to pass the zip function to the enumerate function.
The enumerate function will take the output of the zip function, wrap in into a tuple and associate it with an index value. The ‘for x, value’ part of the statement is assigning names to those values with x representing the index and value representing the tuple.
The final part of the statement ‘{x: {“time”: value[0].text, “name”: value[1].text}’ is pretty much standard dictionary comprehension stuff, were using the values we’ve just retrieved to write out the entries to the dictionary. Note that we access the both the time and name values by dereferencing the tuple using and index value.
The code is not all that complicated and once written it seems pretty obvious but if your just starting out with this stuff it could be intimidating. I hope this example will help someone else trying to accomplish the same task.
