Text clustering/NLP [closed]












1















Imagine there is a column in dataset representing university. We need to classify the values, i.e. number of groups after classification should be as equal as possible to real number of universities. The problem is that there might be different naming for the same university. An example: University of Stanford = Stanford University = Uni of Stanford. Is there any certain NLP method/function/solution in Python 3?



Let's consider both cases: data might be tagged as well as untagged.



Thanks in advance.






python machine-learning nlp text-classification







share|improve this question















closed as too broad by cricket_007, usr2564301, Owen Pauling, jcubic, Janusz Nov 15 '18 at 14:16


Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.



















  • Speaking of Stanford, have you heard of the CoreNLP library? Tried it?

    – cricket_007
    Nov 15 '18 at 7:36













  • I will try it, haven't heard of it. Thanks for the sharing.

    – BC1554
    Nov 17 '18 at 17:16













  • HI, I am on elasticserach, not python so it is kinda different.. I goy yo spend lots of time to find a solution... please find my problem in the comment below.

    – BC1554
    Dec 6 '18 at 6:04











  • Sure, your data is stored there. That doesn't mean you can't query for something, then use Python libraries to do something else to the data, then insert results back to Elastic

    – cricket_007
    Dec 6 '18 at 14:07











  • @cricket_007 thats an option. I'm curious how that would behave if i have 5 milion unique names from elastic and want to match them with original list of 5 thousand unviverisites names. My first idea is to try matching using n-grams (probably trigrams vs. bigrams + stemmed vs. not stemmed). What are your thoughts?

    – BC1554
    Dec 6 '18 at 14:32
















1















Imagine there is a column in dataset representing university. We need to classify the values, i.e. number of groups after classification should be as equal as possible to real number of universities. The problem is that there might be different naming for the same university. An example: University of Stanford = Stanford University = Uni of Stanford. Is there any certain NLP method/function/solution in Python 3?



Let's consider both cases: data might be tagged as well as untagged.



Thanks in advance.






python machine-learning nlp text-classification







share|improve this question















closed as too broad by cricket_007, usr2564301, Owen Pauling, jcubic, Janusz Nov 15 '18 at 14:16


Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.



















  • Speaking of Stanford, have you heard of the CoreNLP library? Tried it?

    – cricket_007
    Nov 15 '18 at 7:36













  • I will try it, haven't heard of it. Thanks for the sharing.

    – BC1554
    Nov 17 '18 at 17:16













  • HI, I am on elasticserach, not python so it is kinda different.. I goy yo spend lots of time to find a solution... please find my problem in the comment below.

    – BC1554
    Dec 6 '18 at 6:04











  • Sure, your data is stored there. That doesn't mean you can't query for something, then use Python libraries to do something else to the data, then insert results back to Elastic

    – cricket_007
    Dec 6 '18 at 14:07











  • @cricket_007 thats an option. I'm curious how that would behave if i have 5 milion unique names from elastic and want to match them with original list of 5 thousand unviverisites names. My first idea is to try matching using n-grams (probably trigrams vs. bigrams + stemmed vs. not stemmed). What are your thoughts?

    – BC1554
    Dec 6 '18 at 14:32














1












1








1








Imagine there is a column in dataset representing university. We need to classify the values, i.e. number of groups after classification should be as equal as possible to real number of universities. The problem is that there might be different naming for the same university. An example: University of Stanford = Stanford University = Uni of Stanford. Is there any certain NLP method/function/solution in Python 3?



Let's consider both cases: data might be tagged as well as untagged.



Thanks in advance.






python machine-learning nlp text-classification







share|improve this question
















Imagine there is a column in dataset representing university. We need to classify the values, i.e. number of groups after classification should be as equal as possible to real number of universities. The problem is that there might be different naming for the same university. An example: University of Stanford = Stanford University = Uni of Stanford. Is there any certain NLP method/function/solution in Python 3?



Let's consider both cases: data might be tagged as well as untagged.



Thanks in advance.






python machine-learning nlp text-classification




python machine-learning nlp text-classification






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 17 '18 at 21:15







BC1554

















asked Nov 15 '18 at 7:22









BC1554BC1554

297




297




closed as too broad by cricket_007, usr2564301, Owen Pauling, jcubic, Janusz Nov 15 '18 at 14:16


Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.









closed as too broad by cricket_007, usr2564301, Owen Pauling, jcubic, Janusz Nov 15 '18 at 14:16


Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.















  • Speaking of Stanford, have you heard of the CoreNLP library? Tried it?

    – cricket_007
    Nov 15 '18 at 7:36













  • I will try it, haven't heard of it. Thanks for the sharing.

    – BC1554
    Nov 17 '18 at 17:16













  • HI, I am on elasticserach, not python so it is kinda different.. I goy yo spend lots of time to find a solution... please find my problem in the comment below.

    – BC1554
    Dec 6 '18 at 6:04











  • Sure, your data is stored there. That doesn't mean you can't query for something, then use Python libraries to do something else to the data, then insert results back to Elastic

    – cricket_007
    Dec 6 '18 at 14:07











  • @cricket_007 thats an option. I'm curious how that would behave if i have 5 milion unique names from elastic and want to match them with original list of 5 thousand unviverisites names. My first idea is to try matching using n-grams (probably trigrams vs. bigrams + stemmed vs. not stemmed). What are your thoughts?

    – BC1554
    Dec 6 '18 at 14:32



















  • Speaking of Stanford, have you heard of the CoreNLP library? Tried it?

    – cricket_007
    Nov 15 '18 at 7:36













  • I will try it, haven't heard of it. Thanks for the sharing.

    – BC1554
    Nov 17 '18 at 17:16













  • HI, I am on elasticserach, not python so it is kinda different.. I goy yo spend lots of time to find a solution... please find my problem in the comment below.

    – BC1554
    Dec 6 '18 at 6:04











  • Sure, your data is stored there. That doesn't mean you can't query for something, then use Python libraries to do something else to the data, then insert results back to Elastic

    – cricket_007
    Dec 6 '18 at 14:07











  • @cricket_007 thats an option. I'm curious how that would behave if i have 5 milion unique names from elastic and want to match them with original list of 5 thousand unviverisites names. My first idea is to try matching using n-grams (probably trigrams vs. bigrams + stemmed vs. not stemmed). What are your thoughts?

    – BC1554
    Dec 6 '18 at 14:32

















Speaking of Stanford, have you heard of the CoreNLP library? Tried it?

– cricket_007
Nov 15 '18 at 7:36







Speaking of Stanford, have you heard of the CoreNLP library? Tried it?

– cricket_007
Nov 15 '18 at 7:36















I will try it, haven't heard of it. Thanks for the sharing.

– BC1554
Nov 17 '18 at 17:16







I will try it, haven't heard of it. Thanks for the sharing.

– BC1554
Nov 17 '18 at 17:16















HI, I am on elasticserach, not python so it is kinda different.. I goy yo spend lots of time to find a solution... please find my problem in the comment below.

– BC1554
Dec 6 '18 at 6:04





HI, I am on elasticserach, not python so it is kinda different.. I goy yo spend lots of time to find a solution... please find my problem in the comment below.

– BC1554
Dec 6 '18 at 6:04













Sure, your data is stored there. That doesn't mean you can't query for something, then use Python libraries to do something else to the data, then insert results back to Elastic

– cricket_007
Dec 6 '18 at 14:07





Sure, your data is stored there. That doesn't mean you can't query for something, then use Python libraries to do something else to the data, then insert results back to Elastic

– cricket_007
Dec 6 '18 at 14:07













@cricket_007 thats an option. I'm curious how that would behave if i have 5 milion unique names from elastic and want to match them with original list of 5 thousand unviverisites names. My first idea is to try matching using n-grams (probably trigrams vs. bigrams + stemmed vs. not stemmed). What are your thoughts?

– BC1554
Dec 6 '18 at 14:32





@cricket_007 thats an option. I'm curious how that would behave if i have 5 milion unique names from elastic and want to match them with original list of 5 thousand unviverisites names. My first idea is to try matching using n-grams (probably trigrams vs. bigrams + stemmed vs. not stemmed). What are your thoughts?

– BC1554
Dec 6 '18 at 14:32












1 Answer
1






active

oldest

votes


















2














A very simple unsupervised approach would be to use a k-means based approach. The advantage here is that you know exactly how many clusters (k) you expect, since you know the number of universities in advance.



Then you could use a package such as scikit-learn to create your feature vectors (most likely n-grams of characters using a Countvectorizer with the option analyzer=char) and you can use the clustering to group together similarly written universities.



There is no guarantee that the groups will match perfectly, but I think that it should work quite well, as long as the different spellings are somewhat similar.






share|improve this answer
























  • @BC1554 Any update on whether this approach was useful?

    – Ivo Merchiers
    Nov 30 '18 at 8:55











  • Hi I thought I will use Python in my new job..However... I am using ElasticSearch so it is kind of different. I am trying to implement match_phrase + fuzziness but it seems like it is impossible (not that much examples online, no such cases in the documentation describer)... Anyboday has experience on phrase matching including fuzziness on ElasticSearch? Thanks in advance :)

    – BC1554
    Dec 6 '18 at 6:03


















1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









2














A very simple unsupervised approach would be to use a k-means based approach. The advantage here is that you know exactly how many clusters (k) you expect, since you know the number of universities in advance.



Then you could use a package such as scikit-learn to create your feature vectors (most likely n-grams of characters using a Countvectorizer with the option analyzer=char) and you can use the clustering to group together similarly written universities.



There is no guarantee that the groups will match perfectly, but I think that it should work quite well, as long as the different spellings are somewhat similar.






share|improve this answer
























  • @BC1554 Any update on whether this approach was useful?

    – Ivo Merchiers
    Nov 30 '18 at 8:55











  • Hi I thought I will use Python in my new job..However... I am using ElasticSearch so it is kind of different. I am trying to implement match_phrase + fuzziness but it seems like it is impossible (not that much examples online, no such cases in the documentation describer)... Anyboday has experience on phrase matching including fuzziness on ElasticSearch? Thanks in advance :)

    – BC1554
    Dec 6 '18 at 6:03
















2














A very simple unsupervised approach would be to use a k-means based approach. The advantage here is that you know exactly how many clusters (k) you expect, since you know the number of universities in advance.



Then you could use a package such as scikit-learn to create your feature vectors (most likely n-grams of characters using a Countvectorizer with the option analyzer=char) and you can use the clustering to group together similarly written universities.



There is no guarantee that the groups will match perfectly, but I think that it should work quite well, as long as the different spellings are somewhat similar.






share|improve this answer
























  • @BC1554 Any update on whether this approach was useful?

    – Ivo Merchiers
    Nov 30 '18 at 8:55











  • Hi I thought I will use Python in my new job..However... I am using ElasticSearch so it is kind of different. I am trying to implement match_phrase + fuzziness but it seems like it is impossible (not that much examples online, no such cases in the documentation describer)... Anyboday has experience on phrase matching including fuzziness on ElasticSearch? Thanks in advance :)

    – BC1554
    Dec 6 '18 at 6:03














2












2








2







A very simple unsupervised approach would be to use a k-means based approach. The advantage here is that you know exactly how many clusters (k) you expect, since you know the number of universities in advance.



Then you could use a package such as scikit-learn to create your feature vectors (most likely n-grams of characters using a Countvectorizer with the option analyzer=char) and you can use the clustering to group together similarly written universities.



There is no guarantee that the groups will match perfectly, but I think that it should work quite well, as long as the different spellings are somewhat similar.






share|improve this answer













A very simple unsupervised approach would be to use a k-means based approach. The advantage here is that you know exactly how many clusters (k) you expect, since you know the number of universities in advance.



Then you could use a package such as scikit-learn to create your feature vectors (most likely n-grams of characters using a Countvectorizer with the option analyzer=char) and you can use the clustering to group together similarly written universities.



There is no guarantee that the groups will match perfectly, but I think that it should work quite well, as long as the different spellings are somewhat similar.







share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 15 '18 at 7:44









Ivo MerchiersIvo Merchiers

643116




643116













  • @BC1554 Any update on whether this approach was useful?

    – Ivo Merchiers
    Nov 30 '18 at 8:55











  • Hi I thought I will use Python in my new job..However... I am using ElasticSearch so it is kind of different. I am trying to implement match_phrase + fuzziness but it seems like it is impossible (not that much examples online, no such cases in the documentation describer)... Anyboday has experience on phrase matching including fuzziness on ElasticSearch? Thanks in advance :)

    – BC1554
    Dec 6 '18 at 6:03



















  • @BC1554 Any update on whether this approach was useful?

    – Ivo Merchiers
    Nov 30 '18 at 8:55











  • Hi I thought I will use Python in my new job..However... I am using ElasticSearch so it is kind of different. I am trying to implement match_phrase + fuzziness but it seems like it is impossible (not that much examples online, no such cases in the documentation describer)... Anyboday has experience on phrase matching including fuzziness on ElasticSearch? Thanks in advance :)

    – BC1554
    Dec 6 '18 at 6:03

















@BC1554 Any update on whether this approach was useful?

– Ivo Merchiers
Nov 30 '18 at 8:55





@BC1554 Any update on whether this approach was useful?

– Ivo Merchiers
Nov 30 '18 at 8:55













Hi I thought I will use Python in my new job..However... I am using ElasticSearch so it is kind of different. I am trying to implement match_phrase + fuzziness but it seems like it is impossible (not that much examples online, no such cases in the documentation describer)... Anyboday has experience on phrase matching including fuzziness on ElasticSearch? Thanks in advance :)

– BC1554
Dec 6 '18 at 6:03





Hi I thought I will use Python in my new job..However... I am using ElasticSearch so it is kind of different. I am trying to implement match_phrase + fuzziness but it seems like it is impossible (not that much examples online, no such cases in the documentation describer)... Anyboday has experience on phrase matching including fuzziness on ElasticSearch? Thanks in advance :)

– BC1554
Dec 6 '18 at 6:03





-

Popular posts from this blog

Florida Star v. B. J. F.

Image
Florida Star v. B. J. F. From Wikipedia, the free encyclopedia Jump to navigation Jump to search United States Supreme Court case Florida Star v. B. J. F. Supreme Court of the United States Argued March 21, 1989 Decided June 21, 1989 Full case name The Florida Star v. B. J. F. Citations 491 U.S. 524 ( more ) 109 S. Ct. 2603; 105 L. Ed. 2d 443; 1989 U.S. LEXIS 3120; 57 U.S.L.W. 4816; 16 Media L. Rep. 1801 Prior history The Florida Star v. B.J.F., 530 So.2d 286 (1988) Supreme Court of Florida; Florida Star v. B.J.F., 499 So.2d 883 (1986) Fla. Dist. Court of Appeals Holding Florida Stat. § 794.03 is unconstitutional to the extent it makes the truthful reporting of information that was a matter of public record unlawful, as it violates the First Amendment. Court membership Chief Justice William Rehnquist Associate Justices William J. Brennan Jr.  · Byron White Thurgood Marshall  · Harry Blac...
Read more

Danny Elfman

Image
Danny Elfman From Wikipedia, the free encyclopedia Jump to navigation Jump to search Danny Elfman Elfman at the 2010 San Diego Comic-Con Born Daniel Robert Elfman ( 1953-05-29 ) May 29, 1953 (age 65) Los Angeles, California, U.S. Spouse(s) Bridget Fonda ( m.  2003) Children 1 Musical career Genres Rock [1] ska [2] new wave film music video game music Occupation(s) Composer, singer, songwriter, record producer Instruments Trombone guitar percussion vocals keyboards [3] Years active 1972–present Associated acts Oingo Boingo James Newton Howard Daniel Robert Elfman (born May 29, 1953) is an American composer, singer, songwriter, and record producer. Elfman first became known for being the lead singer and songwriter for the band Oingo Boingo from 1974 to 1995. He is well known for scoring films and television shows, particularly his frequent collabora...
Read more

Lugert, Oklahoma

Image
Unincorporated community in Oklahoma, United States Lugert, Oklahoma Unincorporated community Foundations from the town of Lugert at the bottom of Lake Altus-Lugert Lugert, Oklahoma Location within the state of Oklahoma Coordinates: 34°53′45″N 99°16′31″W  /  34.89583°N 99.27528°W  / 34.89583; -99.27528 Coordinates: 34°53′45″N 99°16′31″W  /  34.89583°N 99.27528°W  / 34.89583; -99.27528 Country United States State Oklahoma County Kiowa Time zone UTC-6 (Central (CST))  • Summer (DST) UTC-5 (CDT) GNIS feature ID 1100597 Lugert is an unincorporated community in Kiowa County, Oklahoma. [1] The town of Lugert was founded in 1901 on 80 acres (320,000 m 2 ). In the town, there was a general store that housed the Post office and sold dry goods, school supplies, groceries, harnesses, axes and much more. It was named for Frank Lugert, who had moved to the area in 1898. Lugert owned land where the town was sited and also owned the general st...
Read more