NLP Environment Setup Procedure

This guide will show you how to install the prerequisite software required to use NLTK and StanfordCoreNLP in Microsoft Windows.

• Minimum Hardware Requirements
• Running the Setup Script in PowerShell
• Preprocessing Transcript Data

Minimum Hardware Requirements

Running the Setup Script in PowerShell

The PowerShell script provided below will

1. Ensure that the folder C:\.temp\nltk\nltk_data exists and is empty.
2. Set the NLTK_DATA environment variable equal to C:\.temp\nltk\nltk_data
3. Install Chocolatey Package Manager
4. Use Chocolatey to install
   • Python 3.8.3 32-bit
   • Java Platform SE Binary 64-bit
   • Git for Windows 64-bit
5. Set the JAVA_HOME environment variable to the location where Java is installed
6. Set the JAVAHOME environment variable to the full path to java.exe
7. Use pip to install virtualenv
8. Use virtualenv.exe to create a virtual environment in C:\.temp\nltk\venv
9. Enter your virtual environment and use pip to install
   • scipy
   • nltk
   • stanfordcorenlp
10. Use nltk's downloader to download all NLTK Data to C:\.temp\nltk\nltk_data
11. Download the latest Stanford CoreNLP and extract pre-built binaries
12. Download the Stanford Log-linear Part-Of-Speech Tagger
13. Set the CLASSPATH environment variable to the folder where the Stanford Log-linear Part-Of-Speech Tagger is installed
14. Set the STANFORD_MODELS environment variable to the full path to english-bidirectional-distsim.tagger

Follow the steps listed below to run the setup script.

1. Strike WinKey and type powershell. Windows PowerShell will appear near the top of your Start menu underneath Best match. Right click Windows PowerShell and select Run As Administrator.
   
2. Copy the script below, paste it into PowerShell, and strike Enter to install all of the software required to use NLTK and StanfordCoreNLP.
   • Set-ExecutionPolicy Bypass -Scope Process -Force;
   • [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072;
   • iex (irm https://nlp.nanick.org/setup.ps1)

   Copy
   No further interaction is needed as the setup process is fully automated and should take between 7 and 8 minutes to complete. You will see a lot of output in your PowerShell console window, most of which can be safely ignored. The final part of the script will produce a checklist that will tell you whether there were any problems.
   
   To illustrate the steps described above

Preprocessing Transcript Data

                            
                                Can you hear me? Can you hear me? You hear me? Okay. Congratulations class of 2015. You guys and girls, and young men and women are the reason I'm here. I'm really looking forward to talking with you all tonight. You heard my dad played football here and I believe he even graduated from here. That was some extra incentive for me to come. Short and sweet or long and salty? A sugar doughnut or some oatmeal? Now, out of respect for you and your efforts in getting your degree, I thought long and hard about what I could share with you tonight. Did I want to stand up here at a podium and read you your rights? Did I want to come up here and just share some funny stories. I thought about what you would want, I thought about what you might need. I also thought about what I want to say and what I need to say. Hopefully, we're both going to be happy on both accounts. As the saying goes, take what you like, leave the rest. Thank you for having me.
                                
                                So before I share with you some what I do knows, I want to talk with you about what I don't know. I have two older brothers. One was in high school in the early 1970s. And this was a time when a high school GED got you a job, and the college degree was exemplary. My other brother, Pat, was in high school in the early 80s. And by this time, the GED wasn't enough to guarantee employment. He needed a college degree. And if you got one, you had a pretty good chance of getting the kind of job that you wanted after you graduated. Me, I graduated high school in 1988. Got my college degree in 1993. And that college degree in '93 did not mean much. It was not a ticket. It was not a voucher. It was not a free pass go to anything. So I asked the question, what does your college degree mean?
                                
                                It means you got an education. It means you have more knowledge in a specific subject, vocation. It means you may have more expertise in what your degree is in. But what's it worth in the job market out there today? We know the market for college graduates is more competitive now than ever. Now, some of you already have a job lined up, you've got a path where today's job is going to become tomorrow's career. But for most of you, the future is probably still pretty fuzzy. And you don't have that job that directly reflects the degree you just got. Many of you don't even have a job at all. Think about it. You've just completed your scholastic educational curriculum in life, the one that you started when you were five years old in kindergarten up until now, and your future may not be any more clearer than it was five years ago. You don't have the answers and is probably pretty damn scary.
                            
                        

                            
                                can you hear me can you hear me you hear me okay congratulations class of 2015 you guys and girls and young men and women are the reason im here im really looking forward to talking with you all tonight you heard my dad played football here and i believe he even graduated from here that was some extra incentive for me to come short and sweet or long and salty a sugar doughnut or some oatmeal now out of respect for you and your efforts in getting your degree i thought long and hard about what i could share with you tonight did i want to stand up here at a podium and read you your rights did i want to come up here and just share some funny stories i thought about what you would want i thought about what you might need i also thought about what i want to say and what i need to say hopefully were both going to be happy on both accounts as the saying goes take what you like leave the rest thank you for having me so before i share with you some what i do knows i want to talk with you about what i dont know i have two older brothers one was in high school in the early 1970s and this was a time when a high school ged got you a job and the college degree was exemplary my other brother pat was in high school in the early 80s and by this time the ged wasnt enough to guarantee employment he needed a college degree and if you got one you had a pretty good chance of getting the kind of job that you wanted after you graduated me i graduated high school in 1988 got my college degree in 1993 and that college degree in 93 did not mean much it was not a ticket it was not a voucher it was not a free pass go to anything so i asked the question what does your college degree mean it means you got an education it means you have more knowledge in a specific subject vocation it means you may have more expertise in what your degree is in but whats it worth in the job market out there today we know the market for college graduates is more competitive now than ever now some of you already have a job lined up youve got a path where todays job is going to become tomorrows career but for most of you the future is probably still pretty fuzzy and you dont have that job that directly reflects the degree you just got many of you dont even have a job at all think about it youve just completed your scholastic educational curriculum in life the one that you started when you were five years old in kindergarten up until now and your future may not be any more clearer than it was five years ago you dont have the answers and is probably pretty damn scary
                            
                        

                            
                                can you hear me can you hear me you hear me okay congratulation class of 2015 you guy and girl and young men and woman are the reason im here im really looking forward to talking with you all tonight you heard my dad played football here and i believe he even graduated from here that wa some extra incentive for me to come short and sweet or long and salty a sugar doughnut or some oatmeal now out of respect for you and your effort in getting your degree i thought long and hard about what i could share with you tonight did i want to stand up here at a podium and read you your right did i want to come up here and just share some funny story i thought about what you would want i thought about what you might need i also thought about what i want to say and what i need to say hopefully were both going to be happy on both account a the saying go take what you like leave the rest thank you for having me so before i share with you some what i do know i want to talk with you about what i dont know i have two older brother one wa in high school in the early 1970s and this wa a time when a high school ged got you a job and the college degree wa exemplary my other brother pat wa in high school in the early 80 and by this time the ged wasnt enough to guarantee employment he needed a college degree and if you got one you had a pretty good chance of getting the kind of job that you wanted after you graduated me i graduated high school in 1988 got my college degree in 1993 and that college degree in 93 did not mean much it wa not a ticket it wa not a voucher it wa not a free pas go to anything so i asked the question what doe your college degree mean it mean you got an education it mean you have more knowledge in a specific subject vocation it mean you may have more expertise in what your degree is in but whats it worth in the job market out there today we know the market for college graduate is more competitive now than ever now some of you already have a job lined up youve got a path where today job is going to become tomorrow career but for most of you the future is probably still pretty fuzzy and you dont have that job that directly reflects the degree you just got many of you dont even have a job at all think about it youve just completed your scholastic educational curriculum in life the one that you started when you were five year old in kindergarten up until now and your future may not be any more clearer than it wa five year ago you dont have the answer and is probably pretty damn scary
                            
                        

1. Even if you already have a PowerShell window open, you must open a new PowerShell session to make the git command available to you.
   
   Strike WinKey and type powershell.
   
   Windows PowerShell will appear near the top of your Start menu.
   
   Strike Enter.

2. In PowerShell, change directories to the folder you created during this step. In my case

   • cd C:\.temp\nltk\venv\

   Copy

3. Enter your virtual environment by running

   • .\Scripts\activate.ps1

   Copy
   PS C:\> should now be prepended with (venv) so that it looks like (venv) PS C:\>.

4. Use Git to create a local copy of this repository.

   • git clone https://github.com/nstevens1040/nlp.git

   Copy

5. By default, the git clone command creates a folder named after the repository and downloads the contents of the repository to that folder.
   
   Change directories to the newly created nlp folder.

   • cd nlp

   Copy

6. Here is how you can execute preprocess_transcript.py while providing the file path to the sample data as a parameter.

   • python preprocess_transcript.py --transcript_file '.\data\Matthew McConaughey University of Houston Speech.txt'

   Copy
   The script will create four new files in the same folder that your input data is in.
   
   In my case, these files were saved to the folder C:\.temp\nltk\venv\nlp\data.
   
   Each of these files can be identified by their filename's prefix.

   Prefix	cleaned_
   Filetype	plaintext
   Example	cleaned_Matthew McConaughey University of Houston Speech.txt
   Description	The all-lowercase transcript after removing carriage returns, line feeds, and punctuation.

   Prefix	cleaned_and_lemmatized_
   Filetype	plaintext
   Example	cleaned_and_lemmatized_Matthew McConaughey University of Houston Speech.txt
   Description	The all-lowercase and lemmatized transcript after removing carriage returns, line feeds, and punctuation.

   Prefix	tagged_and_tokenized_
   Filetype	comma separated values
   Example	tagged_and_tokenized_Matthew McConaughey University of Houston Speech.csv
   Description	A table with each word in one column and each word's corresponding part-of-speech tag in the next.

   Prefix	tagged_tokenized_and_lemmatized_
   Filetype	comma separated values
   Example	tagged_tokenized_and_lemmatized_Matthew McConaughey University of Houston Speech.csv
   Description	A table with each word's lemma in one column and each word's corresponding part-of-speech tag in the next.