Ndtyjky

Hoping I'm overlooking something stupid here or maybe I don't understand how this is working...

I have an nlp pipeline that does basically the following:

rf_pipeline = Pipeline([

('vect', TfidfVectorizer(tokenizer = spacy_tokenizer)),

('fit', RandomForestClassifier())

])

I run it:

clf = rf_pipeline.fit(X_train, y_train)

preds = clf.predict(X_test)

When I optimize I get accuracy in the high 90's with the following:

confusion_matrix(y_test, preds)

accuracy_score(y_test, preds)

precision_score(y_test, preds)

the TfidfVectorizer is the bottleneck in my computations, so I wanted to break out the pipeline. run the vectorizer, and then do a grid search on the classifier rather than running it on the whole pipeline. Here's how I broke it out:

# initialize

tfidf = TfidfVectorizer(tokenizer = spacy_tokenizer)

# transform and fit

vect = tfidf.fit_transform(X_train)

clf = rf_class.fit(vect, y_train)

# predict

clf.predict(tfidf.fit_transform(X_test))

When I took a look at the accuracy before I ran a full grid search it had plummeted to just over 50%. When I tried increasing the number of trees the score dropped almost 10%.

Any ideas?

For test set, you can't call fit_transform(), but just transform(), otherwise elements of a tfidf vectors has different meaning.

Try this

# predict

clf.predict(tfidf.transform(X_test))