top of page
Search
Writer's pictureVikram Nayyar
Dec 21, 2020

I Created a BOT to COUNT My Blog Views and Graph the Results!!!

Intro:

So in today's post I thought I'd talk about one of my projects and what's cool is that it relates to my blog posts on TheNibbleByte's website!!!😃






It's a program which scrapes my views off the internet and then uses the data to generate a graph showing a trend line of the views. This will be useful for me as I continue to post more and more posts on this website!


I should point out that I'm only scraping from this website and not Medium because I haven't built my audience on there yet.


I should also point out that this solution is by no means perfect. I'm sure that I could've made it more complex, more efficient and added more features however this is just something I've quickly made that does the job (heuristic solution😉).


In this post I will be showing the code (full source code available on my GitHub)


In my opinion, this post isn't really for Python beginners (although it's still worth reading!!). Nevertheless, enjoy!


Imports:

Selenium is an open sourced web-automation tool and the webdriver is what we use to access the page and 'scrape' data.


Time was something I used during development to pause the execution for 10 seconds, just so I knew how long to wait for.


NumPy is a linear algebra library which I used to cleanse the data and convert it from a List to Array.


Pandas is what I used to convert the Array(s) into a DataFrame.


SeaBorn is a Data Visualisation library in which I fed the DataFrame in, as a parameter to create the LinePlot.



Getting the Data:

The first 2 lines are where the chrome WebDriver is used and that's what we use to scrape the web on Chrome: https://chromedriver.chromium.org/downloads .

The 2nd line just creates an instance of the WebDriver, in our Python code which allows us to access the website later.


The next line 'gets' the webpage that contains all my blogposts. By running this line on its own it would just open the webpage and do nothing.


Now I find all the span-tags with the class name "M1M61". This might sound quite random however I found this by using 'inspect element' , looking for the number of views and viewing the HTML:

At this point it should be noted that this step varies depending on your project. The point is that I'm looking for the number of views, this will be handy later.


Next I initialise a Python list called 'views' which will store the view-count for each blog post.


Now I loop through all the view-count's , convert their data type from WebElement to int() and then store them in the list.


The 5 second pause was something I used while developing, it isn't really necessary. Once I've collected the data I then close the page as we're done with it.

Cleansing the Data/Putting it into an Appropriate Format:

It's useless having all the data if it's not in a useful format where we can analyse/interpret it and then plot graphs to visualise it.


At this point, I just want to remind you again that this part isn't the most optimal however it does do the job for the time being!!


When I was looking at the HTML, I noticed that some of the comments had the same class name ("M1M61") as the view count. This is quite challenging because it was different to distinguish between views and comments.


The only way I could distinguish between them is by assuming that comments were never going to be more than 10 (up to date, this has been the case!). Of course in the future, this can cause problems as my number of viewers and comments increase however like I said, it works for now!


To remove the comments values, I looped through the list of 'views' and if the value was <10, I would remove it (basically assuming it was comments, not views). I would also output a message displaying that it was being removed. The word 'None' shouldn't actually be there however that's not a significant issue.


The #print(cleanViews) (commented out) was just going to output the new list without the comments values; again this was purely for development purposes and not required in the solution.




Plotting the Graph:


Graphs need labels so I made a list and appended a label to the list for as many records.


I then converted the labels to an Array and combined them with the actual data to make a graphData DataFrame.


The final line is what actually creates the LinePlot (😂) and it simply calls the lineplot() method, taking the data as a parameter.



Improvements?

No programming project is ever perfect so I thought I'd list some faults/things that could be added.


Maybe one of you could submit a pull request?!?!?!?!?!


See below:

Comments and views need to be distinguished in a clearer way.

Graph could have a different style, more details, stuff like that?

I could make it RealTime (although that would involve deploying to a server (££))?


Final Things:

As always, thank you for making it this far, please leave comments, they help me so much. Be sure to like and share also, lets try and get 100 views within a week, like we have done previously. Follow CompAndCode on all platforms! Thank You Nibble Byte!!!






  • computerscience

  • code

  • coder

  • coding

  • codinglife

  • compandcode

  • computer

  • developer

  • html

  • java

  • javascript

  • machinelearning

  • programmer

  • programming

  • python

  • softwaredeveloper

  • softwareengineer

  • tech

  • technology

  • webdeveloper



  • computerscience

  • code

  • coder

  • coding

  • codinglife

  • compandcode

  • computer

  • developer

  • html

  • java

  • javascript

  • machinelearning

  • programmer

  • programming

  • python

  • softwaredeveloper

  • softwareengineer

  • tech

  • technology

  • webdeveloper


111 views1 comment

Recent Posts

See All

1 comentário

Jamie
21 de dez. de 2020

And I thought your other posts were great, this one was so good! Loved the detail and explanations, would love to see you share more projects that you create!

Curtir
bottom of page