Thursday, August 18, 2016

THE END

It's almost over

Finally after 4 months GSoC the end is near. It was a awesome experience to work with SBML and JSBML team and how much work and efforts they put in this project and what is to work in a collaboration. If it is possible I would like to continue my work after GSoC.

Repository Information

Here is the repository information:
GitHub Commits: https://github.com/hovo1990/deviser/commits/master?author=hovo1990
GitHub Fork Page: https://github.com/hovo1990/deviser

Background

Because there have been of a lot of changes since the first post, I would like to introduce Deviser's code generator to produce Java code once again.

SBML Packages

Computation modeling has become a crucial aspect of biological research, and SBML (the Systems Biology Markup Language) has become the de facto standard open format for exchanging models between software tools in systems biology.
Since SBML Level 3 is being developed as a modular format with optional SBML Level 3 packages are available to extend the focus of the core SBML representation. Deviser is a new code generation system developed to facilitate the development of SBML L3 packages by helping to automate the generation of specifications, UML diagrams and library code.
The current fork extends Deviser's code generator to produce Java code that can be integrated into JSBML, the pure Java API library for SBML. This allows Deviser to target both libSBML and JSBML. At the current stage it generates the foundation of the packages, which facilitates their implementation. 

Using Deviser for Java Code Generation

There is command line version of the function that can be used to invoke the deviser functionality for generating Java/JSBML code from the XML file. This is the deviser.py file found in the generator directory.

deviser.py [--generatejsbml][--latex] input-file.xml  

Requirements for the fork

Deviser requires javap for the java code generation at the current phase. Javap is available with the JDK and the following is required for Devisers' java code functionality.

  1. Java JDK is required for javap
Also set up JAVA_HOME environment variable for your operating system, if you are working under Windows, here is a good tutorial on how to set up. And for Linux: Ubuntu/Debian.

Current State

At the current state Deviser generates compilable templates, which serve as a foundation and facilitate the implementation of the following packages :
  1. Qualitative Models package (qual, for short) allows species in a model to have non-quantitative or non-continuous levels. (Chaouiya et al., 2013)
  2. Groups (groups, Hucka and Smith, 2013) agglomerates SBML model elements and can be linked to annotations and SBO terms (Courtot et al., 2011) to contextualize sets of objects for other programmers and modelers.
  3. Distributions (distrib, Moodie and Smith, 2013) encodes statistical distributions and their sampling.
  4. Dynamic Structures (dyn, Gomez et al., 2014), which supports the definition of dynamical behaviors for model entities.

What is left to be implemented

Specific parts of the packages are left to be implemented.
  1. Flux Balance Constraints (fbc, Olivier and Bergmann, 2013) encodes components for constraints based modeling (Lewis et al., 2012), which employs a class of models in which the canonical stoichiometric relations between reactions and metabolites are specified as constraints for mathematical optimization.
  2. Spatial Processes (spatial, Schaff et al., 2014) specifies geometric descriptions of biochemical models’ components using a cellular coordinate system that can describe non-uniform molecular distributions, diffusive transport and spatially localized reactions.
I would also like to optimize 'imports' parts, as well as to find an alternative to javap, which is used extensively, so deviser does not have any dependency for Java/JSBML code generation.

Java Tests

For running Java code generation tests, please head to generator/tests/test_java_code and run

run_java_tests.py
If there are any errors, it will also show the code differences.

Final Notes

So,this is the end. As I said, it was a great summer and I had a lot of fun working on this project. I could recommend everyone who wants to improve the coding skills to apply for the Google Summer of Code. There's nothing to lose, but you will gain a lot, specially invaluable knowledge and experience. And even if you are not a system biologist, it is a promising field, so checkout the projects of the SBML and JSBML team. 

Acknowledgments

I am very grateful to my mentors Dr. Bergmann, Dr. Keating, Dr. Dräger and Dr. Rodriguez for the invaluable help and patience.

Tuesday, August 16, 2016

Week 17

Hi everyone,

Only one week left. Can't believe that GSoC is almost over.

So what was done during this week:

  1. Important bugs were fixed in readAttributes and toString method generators.
  2. The constructor generator was updated as well as the hashCode generator.
  3. Tests were set up  and run fine both on Windows and Linux.
  4. A prototype README.md version was created.
  5. Updated some parts of the code comments, but still there is work to be done.

The good news is that the parser generator was finalized and that generated code by deviser for qual, dyn,distrib and groups packages  are compilable.

The plan is to spend the next 2-3 days cleaning up the code and updating the comments, since deviser is quite complicated and there are tricky moments. They are available at WIP branch  of the fork repository.

Tuesday, August 9, 2016

Week 16

Hi everyone,

This week the following things were done:

  1. Implemented package-info.java writer
  2. Implemented skeleton for jsbml parser generation.
  3. Updated expand attributes for parser
  4. Added getNamespaceURI() generator
  5. Added getShortLabel() generator
  6. Added isRequired() generator
  7. getPackageName() generator
  8. getPackageNamespaces()
  9. getNamespaces()
  10. getNamespaceFor(int level, int version, int packageVersion)
  11. createPluginFor() method generator.
  12. getListOfSBMLElementsToWrite(Object sbase)
  13. processAttribute() generator
  14. processEndElement() generator
And the good news are that the codes  are generated both under Windows and Linux(430d214).

The only thing left for the parser generation is to finalize processStartElement and writeElement methods.
The unusual order of the "else if" statements generation for processStartElement turned out a little problematic.
The only thing that is challenging is that  package specific features are treated differently in JSBML from libSBML, such is the case for FBCs'     "And" and "Or" treatment.






Wednesday, August 3, 2016

Week 15

Hi everyone,

This week the following things were done:

  1. Implemented enum writer
  2. Devised the JSBML constants writer
  3. Devised the plugins writer
One addition to the tests was creating a new function for comparing the generated files with the test units, the new function uses difflib from python and now it is possible to see the differences and the lines if there are errors.
And the good news are that the codes generated for qual, fbc, dyn, distrib and groups packages are compilable(53dd6b7).

So far from the packages only 'spatial' is left to be adapted, that means the necessary functions need to be adapted for the code generation. And there is also the question for the generation of the parser.

Tuesday, July 19, 2016

Week 13

Still Work in Progress....

Tests

Last week a compilable class goal was achieved.So this week was about setting up tests.
Since deviser generates and select random UUID and prime number, the two methods were adapted for running tests. In the case of prime number, random.seed was set to 0 as for the UUID to a fixed number,as random seed would not affect it.
The foundation of java tests served test_cpp_code files, which were adapted for the java files generation. The path separator was changed to os.sep, so it uses OS specific separator.

javap parser 

jsbml jar was put inside deviser package and javap works under Windows as well.

List of modifications

  1. added lo_element writer
  2. fixed problem with brackets for equals and copy constructor.




Tuesday, July 12, 2016

Week 12

Hello all,

Here's the summary:

Compilable Class

Deviser generates a compilable QualitativeSpecie.java(Fig.1). The class also worked as a replacement in the official jsbml code, but since deviser does not generate not generate   QualitativeSpecies(species) constructor, that part of the code had to be commented in the official jsbml code, after that the compilation was successful(https://gist.github.com/hovo1990/c3ce45eb0d61cefe5c2440d65a45d5be).

Fig.1 Compilable QualitativeSpecies.java

UUID generator

Dr. Bergmann gave valuable help and recommendation for the generation of UUIDs.
Python includes uuid module, that can be used for the generation of uuids, but since Java uses long 64 bit, while python generates 128 bit, the generated uuid had to be masked.
After changing to 56 bit, java compiler would not give more errors for being too long.

hashCode method changes

Generated prime numbers were changed to a global variable, so that the numbers are only generated once.

Implemented Methods

Here's the list of the methods, that were implemented during this week:
  1. prototype writeXMLAttributes
  2. fixed functions with same name generator
  3. prototype toString
  4. prototype readAttribute

Tuesday, July 5, 2016

Week 11

Organize helper functions

Removed duplicate codes and  placed frequent helper functions inside a new module: jsbmlHelperFunctions.py, which includes functions for determining whether a method has to be overriden, javap argument parser, get javadoc comments, finding methods with same name that take different argument and prime number generator.

Constructor generator

Deviser now generates copy constructors.

Method generators

Deviser can generate get_instance, is_set_instance, equals and hashCode methods.

hashCode method

The generator uses algorithm found from stackoverflow.  Right now generates  up to 10 to the 6th power(has to be changed) and than uses pythons random.choice  method for selecting a prime number, but this part needs to be discussed with the mentors. 

Things to do

With the generation of the new methods, previous errors during compilation would vanish, but new ones appeared.Only 3 methods are left to be implemented: readAttribute, writeAttributes and toString.
Finish readAttribute method, which turned out to be a little problematic, write writeAttributes and create the last method generator,which is for toString method.
As for fixing the order of the functions, the idea is to create a dictionary with the function name as key,sort keys and write methods, but that approach would consume more memory, I think it'll be nice to discuss it during TC.






Tuesday, June 28, 2016

Week 10

Class methods detector

This week involved devising a system for automatic override key detection.
For that I came with the idea to create javap parser and look inside import modules and see which methods are abstract.It works, but maybe I overcomplicated things slightly. For the future I think  I will insert all the data from javap for a particular class inside a dictionary. It will change the implementation, but the logic about how to detect override methods won't.
It can also be used for the detection of the missing constructors,that need to be generated as well as get Instance methods.

Mandatory methods generator

This weeks also included the creation of the Mandatory methods generator.
Dr. Bergmann gave a valuable insight about devisers attribute['reqd'] key,which facilitated the detection of return value for the method.Here's the link.

Things to do

Go back to constructor generator.
In the case of QualitativeSpecies.java, the current generator creates setCompartment(java.lang.String) method, but during compilation I get an error(QualitativeSpecies.java:36: error: QualitativeSpecies is not abstract and does not override abstract method setCompartment(Compartment) in CompartmentalizedSBase), which is visible during javap output,so that is something that has to be addressed. Also will take care of the generation of Instance methods.

 

Monday, June 20, 2016

Week 9

Hi everyone,

Unfortunately last week not much was done because I had a exam at the university.I passed it and got an idea.

The idea came to fruition when last week Dr. Dräger mentioned about java reflection and  jython, but I thought there should to be an alternative.

As to me the alternative can be pyjnius, which is a Python module to access Java classes as Python classes using JNI. It has an active community that comes from Android development, easy to use and does not depend on many dependencies.
These are the dependencies.
  1. Java JDK(OpenJDK also works)
  2. Cython and six
  3. Setup JAVA_HOME environment
  4. Setup JRE_HOME environment

Just with a couple of lines it's possible to see all the methods of the class(raw),
but again  it turns out that java reflection is not enough for imports.For example, IDEs' such as Eclipse have their own parser to determine what to import. Still this approach can be used to get instance() methods which are not available in the Deviser xml descriptions.

I think it'll be nice to discuss it during TC.

Tuesday, June 14, 2016

Week 8

This week involved a lot of investigation and figuring out how to use Devisers' key attributes to import modules necessary for JSBML.
Here's the summary what was done during this weekend.
  1. Removed mId and mName from variable generation
  2. Removed Id and Name methods generation from SetGetFunctions.py
  3. Added simple constructor for JSBML
My previous idea to create the jsbml class hierarchy was not sufficient to determine the import modules unfortunately. So I examined again if there's any pattern for Deviser 'type'( here is the Speadsheet), but there is none, that could be used.

Looks like the best approach will be hybrid and one idea is to use attributes' 'name' or 'xml_name', for example if it is 'compartment', then import org.sbml.jsbml.Compartment and org.sbml.jsbml.CompartmentalizedSBase or if it is 'reaction' then import org.sbml.jsbml.Reaction and associated modules.
The idea is to create a dictionary with the package name and attribute name and to use previous tree structure to detect if it is an interface or not, but this part needs to be discussed with the mentors.





Tuesday, June 7, 2016

Week 7

This week was very interesting, but also a challenging one.

First, the summary of what was done during this weekend.
  1. Added JSBML license writer
  2. Added write_jsbml_types_doc
  3. Opening and closing brackets were modified,so now it looks like Java code.
  4. Code for writing import statements and class header was written.
  5. In SetGetFunctions.py, unset was fixed, now throws "firePropertyChange".
But the biggest challenge was how is deviser supposed to decide which modules to import based on the attributes.

At first, the idea was to create a method similar to expand_attributes and expands_class, but soon found out it wouldn't suffice and there would a lot of problems on how to import.


Fig.1 JSBML Full Type Hierarchy
Thus I decided to use a tree structure to store the Full type Hierarchy of JSBML. Next step was to decide, whether to create my own tree structure or to use template. In the end I decided to use a template, but how without importing modules that are not available in Python and I found a gist summary that uses pythons' collections module, which includes High-performance container datatypes.

As for the keys, decided to use this ones:
['name'] = name of the Class or Interface
['hasParent']  = True or False
['parentNode'] = key name of the parent node
['hasChildren'] =True or False
['childrenNodes'] =List or None
['isInterface'] =True or False
['parentInterfaces'] = List or None
['childrenInterfaces'] = List or None
['isUniqueJSBML'] =True or False (if True, that means only available in JSBML)
['level'] =integer (to describle on what level)
['libSBML_analogue']  if there's analogue in libSBML

Here's the full implementation of the tree.
Next step is to figure out how to import only necessary modules.

Tuesday, May 31, 2016

Week 6

Good news, over the last weekend I got some of the primary parts, mainly all the changes were done in the SetGetFunctions.py and it already generates code. Here are some snippets.

Here are the same functions from QualitativeSpecies.java: official JSBML example  and Deviser generated code.


But there were way more changes and other changes have to be done as well.

  1. Refactoring of the variables and class names has been performed.
  2. In BaseJavaFile.py: It was modified, but the main changes were done to the expand_attributes method, also a new dictionary value was added the "JClassType" for attributes. Since Java has both the primitive and the Class. Example int and Integer.
  3. Major changes in the SetGetFunctions.py, that included modifying write_set,write_get, write_is_set, write_unset and their submodules.
  4. All the changes were pushed to the fork.


Monday, May 23, 2016

Week 5

This week involved understanding the flow of the Deviser package, which included setting up many breakpoints for debugging in PyCharm.

Modules from code_files folder were copied and renamed to java_code_files. The idea is to reuse as much code as possible, also refactoring was performed using PyCharm.

JSBML repository was cloned from GitHub, as well as the fork  was updated to match with the master branch.

Next step will be to modify the SetGetFunctions, which is responsible for the attributes(the package based on the attributes generates the constructors as well as functions)as well as BaseJavaFile.

Monday, May 16, 2016

Week 4

This week was an interesting one.

It involved reading and comparing between the Deviser generated packages with the JSBML examples as well as delving in the references of C++, Java, JSBML.The work was done on the 5 packages(fbc, dyn, distrib,spatial,qual).
The diagrams for the JSBML examples were obtained through ObjectAid in Eclipse.

High resolution PDF links are posted.
Here's the meaning behind the coloured circles:

Green - similar in both LibSBML and JSBML
Blue - similar, but in a different location
Orange - similar, but with some differences
Red - not available in LibSBML

FBC package

dyn package

distrib

spatial

qual

The major differences are observed in the qual package.


After that to have a better understanding of Deviser, which is written in Python, having a  class diagram would be helpful.In the end I installed pylint, which includes pyreverse and here's a PDF diagram,but need to have thorough look at the options.
Deviser classes

On a final note, the fork of the Deviser generates  the JSBML package folder structure as described in the JSBML Manual.


Monday, May 9, 2016

Week 3


Hi everyone!

There were couple of problems, that had to be solved.
Debugging  with Spyder was not as effective, nor comfortable for big project like this, so I set up PyCharm Community Edition, which has very robust debugging tools.
Creating UML diagrams for the SBML and JSBML extensions by hand was not effective and because of that I decided to find programs for the automatic generation of class diagrams and looked through programs such as Umbrello, ArgoUml, SmartUML and others, but in the end I installed ObjectAid for Eclipse, and Oovaide(https://github.com/animatedb/oovaide)
Here's the summary what was done during this weekend.

  1. Migrated from Spyder to PyCharm, Spyders' debugging tools are not as powerful as PyCharm.
  2. Eclipse was set up and all JSBML tests were passed.(Fig.1) 
  3. ObjectAid for Eclipse was set up for the automatic creation of  java class diagrams to aid analysis and comparison between SBML and JSBML extensions. (Fig.2)
  4. Oovaide was set up for the automatic creation of  c++ class diagrams, but it's not as powerful as ObjectAid.
    Fig.1. Passed JSBML tests.



















                                                  
Fig.2 ObjectAid created diagram for FBC extension


Sunday, May 1, 2016

Week 2


Hello all!

GSoC has been going on for a week, and the SBML and JSBML team and I have been getting to know one another through dozens of emails. Since this was my first time working with a highly established project such as Deviser and JSBML, there was a definite learning curve. Dr. Bergmann, Dr. Keating, Dr. Dräger and the rest of the team were incredibly patient with me.

So here's the summary what was done last week:

  1. Forked Deviser
  2. Added placeholder code for the generation of JSBML code.
  3. Created a UML diagramm of the Deviser's SBML code generation to see the interaction between the modules.
     
So next step will include:

  1. Analysis of the modules to see what can be reused
  2. The generation of the filestructure compatible with the JSBML
  3. Creation of the BaseJavaFile

Tuesday, April 26, 2016

Introduction

Hello all!

        My name is Hovakim Grabski. I am Ph.D. student at the department of Medical Biochemistry and Biotechnology at Russian-Armenian (Slavonic) University.I hold a specialist's degree in Bioengineering and Bioinformatics from the Russian-Armenian (Slavonic) University. This summer, I am being mentored by Dr.Frank T, Bergmann and  Dr. Andreas Dräger, both part of the NRNB. My project  focuses on  Deviser,which  is   a   new   code   generation   system   being   developed to facilitate   the   development   of  SBML   L3   packages   by   helping   to   automate   the generation   of   specifications,   UML   diagrams   and   library   code   as   well   as   to   provide   a fast,   efficient   means   of   changing   both   API   code   and   specification   documentation   to facilitate   the   update   of   documentation   and   implementation   necessary   as   the   package itself   evolves.   The   code   generator   currently   targets   ​ libSBML   only and  the   goal   is   to extend   Deviser's   code   generator   to   produce   Java   code   that   can   be   integrated   into JSBML​,   the   pure   Java   API   library   for   SBML.   This   will   allow   Deviser   to   target   both libSBML   and   JSBML​ (​https://github.com/sbmlteam/deviser​)



I am very enthusiastic about the opportunity to work with Dr. Bergmann  and Dr. Dräger and the rest of the NRNB and SBML team. I am sure that it will be a great summer.