Python Implementation of textcat training example

[jpmorris]

I can't find an example of how to train a textcat model (using the config.cfg even better) using the python API and NOT the CLI. Why do I want to avoid the CLI? Because my data is in mongodb. I'd prefer not to create *.spacy files just to do the training (making format from json). I'd rather read and shape from mongdb, train the model and save the model to disk, but write inferences back to mongodb. But all the examples I see are only CLI (with the documentation) I can't find examples of training in python anywhere (I've looked at the code examples and I don't see any decent example). Am I missing some documentation sample code somewehre?

[CodeCheetah]

Hi, you could import these directly in the code as follows...

from spacy.util import load_config
from spacy.cli.init_config import init_config
from spacy.cli.train import train
from spacy.cli.pretrain import pretrain
from spacy.cli.evaluate import evaluate

and then define your own definitions...

def train_model(config: str, output_path: str, ngram_size: int, dropout: float, learn_rate: float, threshold: float):
    """
    Train model with parameters
    """
    train(config, output_path, overrides={"paths.train": "./data/train.spacy",
                                    "paths.dev": "./data/dev.spacy",
                                    "components.textcat_multilabel.model.linear_model.ngram_size": ngram_size,
                                    "training.dropout": dropout,
                                    "training.optimizer.learn_rate": learn_rate,
                                    "components.textcat_multilabel.threshold": threshold
                                    })

As far as binary .spacy files are concerned, cli train() command reads config files with references to .spacy binary files, there are no overrides.

I hope this helps.

[jpmorris]

Thanks this does help. So I don't want to use the CLI (for any portion if I can avoid it). So the more I look at it I think I might want to use spacy.pipeline.texcat . I think the example snippets in https://spacy.io/api/textcategorizer do give a rough guide as to how to train from the python API. (It could be clearer with a full example and 'initialize' for 'train' is confusing.) Then again, the example project given in the exampels does bootstrap off of spacy.cli :
https://github.com/explosion/projects/blob/v3/pipelines/textcat_demo/test_project_textcat_demo.py

EDIT: looking at that example in more detail it appears that it A) converts .jsonl into .spacy files and
B) it defines jobs in the project.yml file which reads those files and interfaces with the CLI. I'm trying to avoid making spacy files. My data is in the database and I want to just pass examples to the train portion of spacy via the Python API and save the model to disk. At inference-time I want to read that model and score in my database. I'd like to avoid the intermediate step of creating .spacy files just so I can use the CLI (or a CLI interface)