How to integrate your own data with a local copy of FlyMine

This page explains how to set up a local copy of the latest FlyMine release and integrate your own data. You will be able to query the combined data in a local web application or with the query API. The instructions here assume you need to add classes and fields to the data model. If the data you want to load is already described by the model you can follow the standard mirror instructions and load your data in at the end.

1. Ensure you have the required software installed and configured - see: prerequisites.
2. Get the InterMine software
3. Create PostgreSQL databases requried. These correspond to database names in a flymine.properties file in your home directory.

   > createdb flymine_xx
   > createdb common-tgt-items
   > createdb common-src-items
   

   If you intend to use the web application:

   > createdb userprofile-flymine_xx
   

4. Create and modify necessary files for new source(s) - see: creating a new source. Include the *_additions.xml files in flymine/dbmodel/build.xml.
5. Create a blank database schema that will include your additions to the data model. In flymine/dbmodel run:

   > ant clean build-db
   

   You should see table/columns corresponding to the new classes you added to the model in the production database.
6. Some tables and indexes need to be added to the production database:

   CREATE TABLE precompute_index (
          name character varying(255),
          "statement" bytea,
         category character varying(255)
   );
   
   CREATE TABLE tracker (
         objectid integer,
         fieldname text,
         sourcename text,
         version integer
   );
   
   CREATE INDEX tracker_objectid ON tracker(objectid);
   

7. Dump the metadata table from the database - this is the table that stores an XML version of the object model. It is needed by any client program that queries the FlyMine. As we have added to the model we need to preserve the copy we just created to restore after the FlyMine data dump has been loaded:

   > pg_dump -c -f updated-metadata-table -t intermine_metadata flymine_xx
   

8. Get the latest data only dump from http://www.flymine.org/download/ (file has -data-only suffix). Load up this dump:

   > zcat dump_file | psql flymine_xx
   

9. This will have overwritten the updated model we created with the previous FlyMine model. Now we need to load back in the updated one:

   > cat updated-metadata-table | psql flymine_xx
   

10. Check you still have all the updated tables/columns in the database and some data. It would be a good idea to make a new dump of the database so you don't have to repeat this procedure again.

    > pg_dump -c -f updated-flymine-version-xx database_name
    

11. Analyse the database to update Postres' statistics:

    > psql flymine_xx
    > set default_statistics_target = 500;
    > analyse;
    

12. You should now be able to load data from your own sources into this database. In flymine integrate run:

    ant -v -Dsource=new_source integrate
    

13. Configure
    • Copy the example flymine.properties and build.properties files to your home directory
    • Edit them to include your own database and tomcat locations and usernames/passwords.
14. Release the web application. In flymine/webapp:

    # the first time only
    > ant build-db-userprofile
    > ant default release-webapp
    

    You only need to run 'build-db-userprofile' the first time you set up a webapp. Use ant remove-webapp if there is already a web application deployed at the specified path:

    # to re-release the webapp
    > ant default remove-webapp release-webapp