Vanilla 4.4 – BiGateway ETL New Features

Dear Readers,

Let’s talk about BiGateway, our Etl master piece, and its new features with Vanilla 4.4. First of all, BiGateway deployment inside companies is moving forward very fast, leading to more and more request from customers. It looks like our product position (in terms of designer interface and in terms of features) is well accepted by etl developers and Bi architects, leading to new demand for extended features (especially from system integrators)

We added some new features (boxes) available from the palet such as :

  • encryption / decryption method at column level (this comes along with existing encryption at file level, available in our workflow), using an custom encryption key
  • Web Service support for data input (and clean WebServices integration : no limitation in terms of xml document structure !!!),
  • New Vanilla-MDM file input, with version control on files. Again a clean integration between 2 modules, at data level

We have extended our support for NoSql database, and fix some minor bugs, to have better integration and better performance with MongoDb and Cassandra database. Funny to notice the number of announcement for support from various editors, in parallel with the number of deployment. On our side, we have now interesting deployment for Vanilla/MongoDB in banks, and Vanilla/Cassandra in retail industry, through partners.

Also, we have now a Weka output, providing a direct extraction from Etl to load ay Weka file format, making easier and smoother the process “extract -> analyse”. It’s a first try, based on a customer request, and no doubt we will extend this “dataming support” to other proprietary file format, in addition with csv support (the most used file format for data mining extraction).

We hope you will enjoy those new features,

Best Regards,


BiGateWay Amazing Features

Dear Readers,

A short note about an amazing feature that can save you valuable time when it comes to have anonymous data. An example is you need to get data from your customers to build Metadata and reports, but he is afraid to have his data out of his office. Our Etl package, BiGateWay, provides an easy to deploy feature to work on your data privacy.

Just use the “Encrypt” function box, and select the column you want to encrypt. You need to define an encryption key (keep it private, it will be asked for decrypting the data)


Here is the result of the output … so easy !


There is another another feature inside Vanilla that can encrypt the full data file, at OS level. Those 2 features, combined, will guarantee that data are secured during transmission between 2 sites, and stay anonymous while the System Integrator is working on it from its office,

Have Fun !

From Business Intelligence to Information Management (part 4)

Dear Readers,

The Vanilla platform provides tools to manage various kind of document : any “standard documents” (office, pdf, xml) , media-oriented documents such as image or video and geo documents (such as geo data and various map formats : Flash maps, kml info, wms geo server …).

To manage those different documents, we provide different studio, which can make complex the first approach of Vanilla when dealing with “more than Bi” :

  • BiGateWay (just refer to previous post of this serie), is able to manage kml document to load maps info into our maps repository (tables)
  • BiWorkflow, provide feature to manage external documents in bulk such as loading/uploading all the documents from a network folder, a (s)ftp or any http(s) server … very convenient to build complex operation … an embeded workflow inside a bi platform !!!
  • Norparena package that take care of any other subjects, such as maps definition, address & building design, but also features to manage document indexation (remember : Vanilla embed Lucene as indeaxtion/search engine)
  • FreeMetrics package, that provides interface to geolocalise Kpi … finally any Bi developer can enjoy a studio to push kpi value on any kind of map, using a visual interface !
  • Eclipse PlugIns are available to make Maps & Kpi available in any Birt reports (flash or html maps) … no need to turn to any commercial package to enjoy maps integration inside your reports !

Maps Visualisation of a set of member measure, from an Olap View

Valuable Information not only stands in reports or database anymore. Blog, Mail, Video, external docs and maps contain rich information that are not yet leverage enough. But managing Information is not only pushing a search engine, intelligent maps, ocr recognition or any data mining process, it’s a global approach that should complies with company objectives. As a platform editor, we have to offer the largest set of integrated tools to help developer to achieve their project.

All those subjects – maps, video, indexation, geo-kpi – go beyond the simple “bi” features. In the next post of the serie, I will explain all thoses features, and how we use it to build complex data visualisation interface, using Vanilla portal, FreeDashboard, Birt, FreeWebReport & FreeAnalysis to display the documents and the data,

Have Fun !


Another note about BiGateWay

Dear Readers,

I received questions about how to use an Olap Structure as a Data Source in a BiGateWay transformation. So this Video, to provide more than simple screenshot this time. In this video, you can see how to define the Data Source (Olap), how to create a simple transformation that take some Olap member item and load them into a new Sql table (its so easy to do !) … and best of bread, how to deploy the transformation and run it from Vanilla portal.

You can access the video from here : BiGateWay Transformation using an Olap Data Source

Don’t confuse : it’s easy to do, but it requieres knowledge or assistance (about Olap & Etl in general, with a focus on Vanilla technologies & modules). Too many times over the past few weeks, I received mail from potential users asking for ponctual assistance … on the final step. Before running 100m in 10 to 12 seconds, people have to learn how to walk, no ?

Have Fun !


A note about BiGateWay

Dear Readers,

BiGateWay, our Data Service module (aka ETL) is getting strong recognition as an integrated piece of software (INTEGRATED is a KEYWORD !). Far away from pure players, both side (Etl & Bi), that try to separate Bi & Etl platforms for commercial reasons, Vanilla is providing an INTEGRATED platform that facilitate development & deployment of “data aware applications”.

Recently, I was delighted to see a complex Etl development where transformation take data coming from various database and load our Kpi database, FreeMetrics. The screenshot below are worth thousand words … BiGateWay is easy to use, easy to start with,

Rh Kpi Measure Integration

Just look at details : loopup, calculation, aggregate, exception handler, surrogate key … all that developer need to build complex data transformation with a visual studio. As part of Vanilla platform, security, database definition, workflow process, scheduler : BiGateWay can use all the Vanilla resources, so you save time in development & deployment setup … and more in production admin !

Have Fun !


From Business Intelligence to Information Management (part 1)

Dear Readers,

Just starting a serie of post to explain how we moved from a full feature Business Intelligence Platform to an Enterprise Information Management Platform. In fact, Vanilla provides all the standard features you can expect to find in a Bi platform : Metadata, Reporting, Olap, Dashboard, Kpi and Maps visualisation, along with additional valuable components embeded into the platform, such as :

– Etl/Workflow/Mdm components : using BiGateWay, BiWorkFlow and Vanilla Mdm, along with Data Profiling / Data quality components, developer have a comprehensive suit of software to manage data

– Document Management / Search Engine : using Vanilla v4, you can now manage any kind of documents (image, external document, video), share those documents using Vanilla portal and start indexation process on various type of text document, to later use an embeded search interface. this comes along with Vanilla document indexation (like reports) and all the tools to manage the library of documents (for content expiration, document authorisation, versioning …) , to provide a global comprehensive document management features

It’s very fun now to compete in some tender (public market, bid, proposal … name it as you want), because we can see customers are moving to “more than only bi” (as we can read through the Big Data marketing noise those past months). In fact, some customers still don’t understand how to align their information management strategy with their internal process, but the process has started : there is a real need to take control on document processing, data flow, and inter application data & document exchange … to better understand what are the data coming from outside, data generated by internal application and data that are published outside the company (public or to partners).

Have Fun !


Vanilla v3 (post 9 on 10) – Compliance to Open Standard

Dear Readers,

Important subject in Open Source area is to be ‘standard compliant’. Bellow is a remember of notions and features we have introduced with Vanilla v2 and enhanced with upcoming Vanilla v3 :jdbc driver, oda drivers and remote call to bi document

Jdbc driver for FreeMetadata, to open FreeMetadata document to any report designer or etl engine that support jdbc databasource.

Note : for Birt report designer, no need to use jdbc driver, but instead, you can use the FreeMetadata plugin. Same situation for BiGateWay Etl : there is a native interface available in BiGateway to manipulate FreeMetadata document (but this support can be usefull for other Etl to access to a FreeMetadata “database alike” structure)

FreeMetadata Jdbc definition in Enterprise Services

Jdbc definition in designer (bellow, our BiGateway Etl)

FreeMetadata – through jdbc request definition – Data visualisation

Oda drivers integration to facilitate data access and hide the complexity of internal storage architecture, and also provide an Eclipse way (like jdbc) to access to many Vanilla resources using oda. We provide different kind of drivers, to access FreeMetadata document, Googole document (spreadsheet or table), inline table or FreeMetrics Kpi database. Bellow is an exemple with a FreeMetrics Kpi Oda driver

Oda drivers list

Kpi data visualisation using FreeDashboard

Oda driver and different kpi views

Result in Vanilla portal

Note : Posting wrong information is a shame – in reference to what I recently saw about a new oda driver to access another Open Source Metadata. All is hard coded in the interface, and they are not abble to publish to the portal … so, what is the use ? What is the credibility of those kind of posts and the credibility of the blog writer ? Using Vanilla, all is smoothly integrated !

External remote call to Vanilla object and Web Services

You can easilly publish any Report or Dashboard document as “public”, using a specific profile (group) for data access. Once publish using Enterprise Services, you can manipulate a Url and embed this Url in any portal. If you have java skills, you can do the same calling our document using a set of Web Services

Url and profile (group) definition using Enteprise Services

Result for copy/paste of Url in a browser

Just have a look at what those features can provide :

– an easy way to use FreeMetadata as a secured central – multi type – database (like RedHat Teiid … but with more designer & security features available)

– easy access to any Vanilla structure using oda drivers, in addition to providing access to others database (Google)

– easy way to embed any Vanilla Bi document in already deployed Web interface (such as Portal, Cms, …)

Have Fun !


BiGateWay – Metadata oriented Etl

Dear Reader,

Around a year after its initial launch, our data service component, Vanilla BiGateway, is reaching a new step and getting important recognition from customers, with application in production, taking care of production data

History of BiGateWay is simple : we wanted to add a “data service” component to our Vanilla platform, with features such as the one developers find in any Etl, but also specific functions to leverage Vanilla documents (such as Olap/FreeAnalysis, Kpi/FreeMetrics or Ldap/Vanilla security replication). So, initially, we did not promote our BiGateWay as a specific Etl platform, because we are offering a complete Bi Platfrom (with Etl embed in it)

Initial result went beyond our expectation, and we got dozens of testers in a couple of months, with always same feedback : “great product, easy to use “. It proved us there was a request for an Etl platform for non-java developer  that respect Etl concept & provide an intuitive interface, along with support from the development team. Just look around & correct me if I’m wrong : I don’t see this configuration (Etl for Bi consultant, Metadata and sql data access – support by dev team – Embeded Bi features) in the Open Source  landscape ?

BiGateWay Designer Interface

Future is bright for BiGateWay :

– On one hand, along with BiWorkflow, we started some deployment in France for application that gather information through forms, manipulate & store those information using Etl features and organise report & dashboard distribution using Vanilla core services. All managed by our BiWorkFlow component.

– On the other hand, now supported multithread transformations, we can compete with pure commercial Etl, even with the comunity version of BiGateWay (that provide at no cost team development, checkIn-CheckOut or other usefull function such as data search & browse).

Have Fun !