captioner package



class captioner.model_zoo.Captioning_Model(params, model_type='Captioning_Model', verbose=1, structure_path=None, weights_path=None, model_name=None, vocabularies=None, store_path=None, set_optimizer=True, clear_dirs=True)

Bases: keras_wrapper.cnn_model.Model_Wrapper

Translation model class. Instance of the Model_Wrapper class (see staged_keras_wrapper).

  • params (dict) – all hyperparameters of the model.
  • model_type (str) – network name type (corresponds to any method defined in the section ‘MODELS’ of this class). Only valid if ‘structure_path’ == None.
  • verbose (int) – split to 0 if you don’t want the model to output informative messages
  • structure_path (str) – path to a Keras’ model json file. If we speficy this parameter then ‘type’ will be only an informative parameter.
  • weights_path (str) – path to the pre-trained weights file (if None, then it will be initialized according to params)
  • model_name (str) – optional name given to the network (if None, then it will be assigned to current time as its name)
  • vocabularies (dict) – vocabularies used for word embedding
  • store_path (str) – path to the folder where the temporal model packups will be stored
  • set_optimizer (bool) – Compile optimizer or not.
  • clear_dirs (bool) – Clean model directories or not.
Video captioning with:
  • Attention mechansim on video frames
  • Conditional LSTM for processing the video
  • Feed forward layers:
    • Context projected to output
    • Last word projected to output
Parameters:params – Dictionary of hyperparmeters.
Neural machine translation consisting in stacking blocks of:
  • Multi-head attention.
  • Dropout.
  • Residual connection.
  • Normalization.
  • Position-wise feed-forward networks.

Positional information is injected to the model via embeddings with positional encoding.

Parameters:params (dict) – Dictionary of params (see

Sets and compiles a new optimizer for the Translation_Model. The configuration is read from Translation_Model.params. :return: None


Set self.params as params. :param params: :return:


alias of captioner.model_zoo.Captioning_Model

captioner.model_zoo.getPositionalEncodingWeights(input_dim, output_dim, name='', verbose=True)

Obtains fixed sinusoidal embeddings for obtaining the positional encoding.

  • input_dim (int) – Input dimension of the embeddings (i.e. vocabulary size).
  • output_dim (int) – Embeddings dimension.
  • name (str) – Name of the layer
  • verbose (int) – Be verbose

A list with sinusoidal embeddings.


Training function. Sets the training parameters from params. Build or loads the model and launches the training. :param params: Dictionary of network hyperparameters. :return: None


captioner.apply_model.sample_ensemble(args, params)

Use several translation models for obtaining predictions from a source text file.

  • args (argparse.Namespace) –

    Arguments given to the method:

    • dataset: Dataset instance with data.
    • text: Text file with source sentences.
    • splits: Splits to sample. Should be already included in the dataset object.
    • dest: Output file to save scores.
    • weights: Weight given to each model in the ensemble. You should provide the same number of weights than models. By default, it applies the same weight to each model (1/N).
    • n_best: Write n-best list (n = beam size).
    • config: Config .pkl for loading the model configuration. If not specified, hyperparameters are read from
    • models: Path to the models.
    • verbose: Be verbose or not.
  • params – parameters of the translation model.


captioner.build_callbacks.buildCallbacks(params, model, dataset)
Builds the selected split of callbacks run during the training of the model:
  • PrintPerformanceMetricOnEpochEndOrEachNUpdates: Evaluates the model in the validation split given a number of epochs/updates.
  • SampleEachNUpdates: Shows several translation samples during training.
  • params (dict) – Dictionary of network hyperparameters.
  • model (Model_Wrapper) – Model instance on which to apply the callback.
  • dataset (Dataset) – Dataset instance on which to apply the callback.

list of callbacks to pass to the Keras’ training.

Module contents