Google Reviews Project¶

The dataset used in this project was scrapped from google maps reviews

IMPORT LIBRARIES¶

LOAD DATA¶

DATASET OVERVIEW¶

Define function to get an overview of the data

Define function to have an overview of the variables

SUMMARY OF STATISTICS¶

Compute summary of statistics for the numerical columns in the DataFrame.

Compute summary of statistics for the categorical columns in the DataFrame

Clean the dataset¶

user_tot_reviews column¶

The ‘user_tot_reviews’ column of our dataset contains a mix of data types, including integers, strings, and NaN values. However, for our current project, we only require the numerical values from this column. We can apply a regular expression pattern to extract only the digits. Once we have successfully extracted the numerical values, we will then convert the data type from objects to integers. This conversion is important as it allows us to perform numerical operations on the data and to facilitate further analysis. By performing this data type conversion, we ensure that the ‘user_tot_reviews’ column contains only numerical values and is in a format that can be easily used in our analysis.

To transform the column dtype to int for analysis purposes, we need to replace empty strings with NaN using the replace method. Since the column dtype is an object, all rows with NaN values were considered as strings and therefore returned no numbers, resulting in empty cells.

date_review column¶

we will first rename it to date_review then we will extract the number of days since the review and then transform the column date type into int for analysis purpose

review_liked¶

This column refers to whether the review received any likes from other users. If the value is “like,” it means that the review did not receive any likes from users. If the value is “nan,” it means that the review received at least one like from a user. To avoid confusion with the “like” value, we will convert “like” to “no” and “nan” to “yes.”

response_date¶

we will first rename it to days_since_response then we will extract the number of days since the response and then transform the column date type into int for analysis purpose

Add a new column named cleaned to fill the nan values in text which refers to ratings with no comment

REVIEW TEXT PROCESSING¶

Tokenization¶

In natural language processing (NLP), the process of removing punctuation marks from text is called “tokenization” or “text normalization”. It involves removing all punctuation marks from the text and converting the text to lowercase.

Expand Contractions¶

Contractions are the shortened versions of words like don’t for do not and how’ll for how will. These are used to reduce the speaking and writing time of words. We need to expand these contractions for a better analysis of the reviews.

We will use a dictionary of common English contractions that for mapping the contractions to their expanded forms:

Lowercase the reviews¶

Natural Language Processing (NLP) models process text as sequences of characters or words to understand the meaning behind them. However, in many cases, different forms of the same word can have different meanings, such as “buy” and “Buy.” This can lead to errors in the analysis, as the models may treat these forms as separate entities.

To overcome this problem, one common solution is to convert all words to lowercase. This ensures that words with different capitalization are treated the same way by the NLP models. For example, “buy” and “Buy” would both be converted to “buy.”

In Python, the lower() function can be used to convert a string to lowercase. This function is commonly used in NLP preprocessing to standardize the capitalization of text before feeding it to an NLP model.

Remove digits and words containing digits¶

When processing text data for Natural Language Processing (NLP) tasks, it is often necessary to clean the text by removing certain types of words or characters that are not useful for the analysis. One common type of unwanted text is numbers and words that contain digits, as they typically do not contribute much to the meaning of the text.

To remove these unwanted elements, regular expressions can be used in conjunction with lambda functions. Regular expressions are patterns that can be used to match specific types of text, while lambda functions are anonymous functions that can be used to apply operations to the text.

In this case, the lambda function is used to apply the regular expression pattern that matches any word containing a digit or any standalone digit in the text data. The re module in Python provides various regular expression functions and methods to work with regular expressions. By using this combination of tools, unwanted numbers and words containing digits can be removed from the text data, which can help improve the accuracy of NLP models.

• df[‘cleaned’]: This selects the ‘cleaned’ column of the DataFrame df.
• .apply(lambda x: …): This applies a function to each element of the ‘cleaned’ column. The function in this case is a lambda function.
• re.sub(‘\w\d\w‘,”, x): This is the lambda function that replaces any alphanumeric character that contains a digit with an empty string.
• The regular expression \w\d\w matches any sequence of characters that includes at least one digit.
• The re.sub() function replaces any match with an empty string, effectively removing the match from the string.

Remove Punctuations¶

Punctuation marks are an essential part of English grammar as they help us to express ourselves clearly and effectively. However, when it comes to text analysis, punctuations can actually hinder our understanding of the text. Punctuations such as commas, hyphens, full stops, and quotation marks, among others, can create noise in our text data that is not useful for analysis. For example, if we want to perform text analysis to identify the most commonly used words in a text, it would be unhelpful to include punctuation marks such as full stops and commas in our analysis. Removing punctuations from text data not only simplifies the text but also ensures that our analysis is accurate and meaningful. Therefore, it is crucial to remove punctuations from text data before performing text analysis.

• df[‘cleaned’]: This selects the ‘cleaned’ column of the DataFrame df.
• .apply(lambda x: …): This applies a function to each element of the ‘cleaned’ column. The function in this case is a lambda function.
• re.sub(‘[%s]’ % re.escape(string.punctuation), ”, x): This is the lambda function that removes all punctuation marks from the input string.
• The regular expression […] matches any single character that is included inside the square brackets. In this case, it includes all the punctuation marks in the string.punctuation string.
• The re.escape() function is used to escape any characters in string.punctuation that have special meaning in regular expressions.
• The re.sub() function replaces all punctuation marks in the input string x with an empty string ”

remove special characters¶

The regular expression r'[^a-zA-z0-9]’ is a pattern that matches any character that is not a letter (upper or lower case) or a digit. Here’s a breakdown of the pattern:

[^…] is a negated character set, which matches any character that is not in the set of characters inside the square brackets. a-zA-Z is a range of characters that includes all upper and lowercase letters of the English alphabet. 0-9 is a range of characters that includes all digits. So the pattern [^a-zA-z0-9] matches any character that is not a letter or a digit, including whitespace, punctuation marks, and other special characters.

Removing extra spaces¶

replace multiple consecutive spaces in the ‘cleaned’ column of a pandas DataFrame with a single space.

• df[‘cleaned’]: This selects the ‘cleaned’ column of the DataFrame df.
• .apply(lambda x: …): This applies a function to each element of the ‘cleaned’ column. The function in this case is a lambda function.
• re.sub(‘ +’,’ ‘,x): This is the lambda function that replaces multiple consecutive spaces in the input string x with a single space.
• The regular expression ‘ +’ matches one or more spaces in the input string. The re.sub() function replaces any match with a single space ‘ ‘. This way, it reduces multiple consecutive spaces in the input string to a single space.

Removal of stopwords and lemmatization¶

The removal of stopwords and lemmatization are common techniques used in natural language processing to preprocess text data for analysis.

Stopwords are words that occur frequently in a language but do not carry much meaning on their own (e.g., “the”, “a”, “an”, “in”, etc.). Removing stopwords from text data can reduce noise and improve the quality of analysis.

Lemmatization is the process of reducing words to their base form (called the “lemma”), which can help to group together variations of the same word (e.g., “run”, “running”, “ran” are all reduced to the lemma “run”). This can reduce the vocabulary size and improve the accuracy of analysis. We’ll use SpaCy for the removal of stopwords and lemmatization.

EXTRACT ONLY ADJECTIVES¶

This function extract_adjectives() takes a string of text as input, tokenizes the text into individual words, tags the parts of speech for each word using the NLTK’s POS tagger, extracts the adjectives from the tagged text, and returns a string of adjectives separated by spaces.

• The input text is tokenized into individual words using the word_tokenize() function from the NLTK library.
• The NLTK’s POS tagger is used to tag the parts of speech for each word in the text. This is done using the pos_tag() function from the NLTK library. The output of this function is a list of tuples, where each tuple contains a word and its associated part of speech tag.
• The adjectives are extracted from the tagged text by iterating over the list of tuples and selecting only those words that have a part of speech tag that starts with ‘JJ’ (which indicates an adjective). The selected words are then joined together into a single string separated by spaces using the join() function.
• Finally, the function returns the string of adjectives.

10 Random reviews after Removal of Punctuations and Numbers¶

SENTIMENT ANALYSIS¶

Sentiment analysis is the analysis of how much a piece of text is positive and opinionated.

Here, in this case, checking only the polarity, i.e., how much a text is positive or negative, is sufficient. We can check the polarity of a review using the TextBlob library in Python:

This code imports the TextBlob library and calculates the polarity value for each row of the ‘cleaned’ column in the DataFrame.

The TextBlob class is a part of the textblob library, which is a Python library for processing textual data. It provides various natural language processing tools, such as sentiment analysis, part-of-speech tagging, and noun phrase extraction.

Text Subjectivity¶

In natural language, subjectivity refers to expression of opinions, evaluations, feelings, and speculations and thus incorporates sentiment. Subjective text is further classified with sentiment or polarity.

READING TIME¶

VISUALIZATION AND REPORTING¶

For reporting purpose we will create another dataset which contains reviews only

Create a Document Term Matrix.¶