Week from 05/08 to 05/15

Start of the week

The task given for this week is to implement a python application for machine learning trained models. For that, I used KBox(https://github.com/AKSW/KBox) as the base. I created an interface between KBox and airML (https://github.com/AKSW/airML) including useful functions for my GSoC project. 

Why?

That is very useful for reproducibility research, it allows other users to reuse models without having to re-train them. For now, I have trained the monument_300 dataset in NSPM.

How?

Instead of reimplementing everything from KBox, I implemented useful functions that allow me to share, locate and install trained models. These following functions are implemented by me:


list(kns=False)
Description:List all available models(kns=False) or list all KNS services(kns=True).
Args:
  kns:'boolean',defines whether to list only the KNS services or not
Returns:
    None
Throws:
    OSError
install(modelID, format=None, version=None)
Description:Install the a model by given modelID
Args:
    modelID: 'string', url of the model hosted in a public repository.
    format:  'string', format of the model.
    version: 'string' specific version to be installed of the the model.
Returns:
    None
Throws:
    OSError
Example:
    install("http://github.org/aksw/NSpM/monument_300","NSPM/Model","0")
install(modelID, format=None, version=None, kns=None)
Description:Install a given model base using the available KNS services to resolve it.
Args:
    modelID:'string',url of the model to be installed.
    format: 'string',format of the model.
    version:'string',version of the model.
    kns:'string', url of the kns service.
Returns:
    None
Throws:
    OSError
removeKNS(kns)
Description:Remove a given KNS service.
Args:
    kns:'string', url of the kns service.
Returns:
    None
Throws:
    OSError
getInfo(model, format=None, version=None)
Description:Gives the information about a specific model.
Args:
    model: url of the model to be installed.
    format: format of the model.
    version: version of the model.
Return:
    None
Throws:
    OSError
locate(modelID, format=None, version=None)
Description:Find the local address of the given model.
Args:
    modelID: 'string', url of the model to be located.
    format: 'string', format of the model.
    version: 'string', version of the model
Returns:
     None
Throws:
    OSError
search(pattern, format=None, version=None)
Description:Search for all model-ids containing a given pattern.
Args:
    pattern: 'string',pattern of the url of the models.
    format: 'string',format of the model.
    version: 'string',version of the model.
Returns:
    None
Throws:
    OSError
getModelDirPath()
Description:Show the path to the folder which contains the models.
Returns:
    Path of the installed models.
Throws:
    OSError
"""
setModelDirPath(dir)
Description:Change the path of the resource folder.
Args:
    dir:'string', new model path
Returns:
    None
Throws:
    OSError
"""
showVersion()
Description:Returns KBox version.
Returns:
    KBox version.
Throws:
    OSError


After implementing the above functions, I trained the monument_300 dataset in NSpM. Then I serialized the trained model and share it in http://figshare.com/. The model link is  https://figshare.com/articles/monument_300_model_zip/12303242.
And I installed the trained model using airML. 

Problems

During the activity, I faced some challenges.  
  • How to port arguments from Java to Python?
  • How to extract stdout from KBox jar?
  • How to distribute a Python package?

How did I solve the problems?

  • I solve the first problem through explicit system calls.
  • By using subprocess module in python. It allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. 
  • Used setuptools library to create and distribute the package.

Comments

Post a Comment

Popular posts from this blog

DBpedia Neural Multilingual QA - GSoC project summery

 GSoC blog This blog includes a detailed description of the work I carried out during the GSoC period. The above mentioned blog includes the tasks carried out on a weekly basis. Brief Description of my work Implement airML using KBox and pip.  This task was to create and distribute KBox  into a pip package called airML, which will allow users to share and dereference ML models Share the Monument dataset with airML. Train the monument dataset and place it in a public repository and dereference it with airML. Create the language detector dataset. Iterate over all questions in IRbench , creating the dataset. Create a language detector model and train it. After creating the model, I evaluated other existing language models and wrote a research paper with my mentor. Experimentation in Machine Translation methods. Creating datasets. Annotating Datasets. Evaluating different methods. The full description of Tasks carried out and the obtained results are included in this blog . The results ob
Read more

Week from 07/12 to 07/18

 Since we are writing a research paper on "Assessing Language Identification Methods and Frameworks over Linked Data", I was tasked to do some work regarding the paper this week. These are the work done in this week, Created a readme file including the results obtained so far the research. The complete results can be found in this Github repo .   Wrote the evaluation part for the research paper including how the experimentation is done. Refactored the code and put a PR to the main repo.
Read more

Machine Translation Task

 This experimentation was done as follows, Dataset Creation using QALD datasets. Train and evaluate the initial dataset on Tensorflow Neural Machine translation with attention  model . Using  DBpedia Spotlight  to annotate the entities and create templates. Manually evaluate the entity recognition process. Train and evaluate the correctly annotated text on Tensorflow Neural Machine translation with attention  model . 1. Dataset Creation using QALD datasets. From the QALD datasets from QALD-3 to QALD-7, I created a dataset which consists of language pairs such as English-Spanish, English-Deutsch, etc. These pairs are created for all languages Deutsch, Spanish, French Brazilian Portuguese, Portuguese, Italian, Dutch, Hindi, Romanian, Persian, and Russian. 2.  Train and evaluate the initial dataset on Tensorflow Neural Machine translation with attention model. Using the " Tensorflow Neural Machine translation with attention model ", trained the initial datasets created as said a
Read more