Delphi now provides a new component based DataSnap server technology.  Existing applications can host their RemoteDataModules inside this new DataSnap server.  Our new DataSnap server does not support the DCOM technology.  If you still need DCOM, the older component set for DataSnap is still included in the product.  For the new DataSnap server, JSON/RPC based messaging layer and pluggable transport layer is provided.

In addition to supporting existing RemoteDataModules, we now provide support for server methods and server connections.  Server methods are our light weight mechanism for remote method invocation.  They are ideal for writing stored procedure logic in an object oriented language such as Object Pascal.  Server methods can be invoked from dbExpress or ADO.NET clients in the same way that stored procedures are.  This is a very simple, easy to use mechanism for remote method invocation.  The DataSnap server uses dynamic method invocation to execute server methods.  This approach allows simple methods on plain old objects to be called from remote clients.  There is no need for a proprietary interface definition language or complicated tools to generate stubs and proxies.  Some developers have expressed interest in statically typed client side proxies for server classes.  These are not difficult to generate and support for this will be provided.  In the initial Tiburon release we may have to provide this capability as a separate example utility due to time constraints.  Tiburon already provides support to collect all the class/method/parameter metadata required to generate proxies allowing us to at least create an example utility to generate proxies.  Note that the more “dynamic” execution of a command by name described in more detail below and a “static” proxy approach are both viable.   There are trade offs to both approaches.  The dynamic approach blends in well with applications that work more directly with SQL queries and stored procedures.  The static approach provides better support for things like code insight and client side type safety when server side methods are refactored.

Server connections allow a client and server application to share a server side dbExpress database connection.  Only the DBXClient driver is needed on the client to access any dbExpress driver installed on the server.  The DBXClient is 100% Object Pascal making for a simple client side deployment.  Server connections allow a client to share the same database connection and transactional context with a server application.  An interesting application of server connections is to allow the client to perform select queries to read data, and to call server methods that use the same server connection to perform all insert, update and delete operations on the data.

The new DataSnap components are implemented in Object Pascal.  We currently provide win32 server side applications.  On the client we support both win32 and .net applications.  Server methods and server connections can both be accessed through both our dbExpress and ADO.NET connectivity solutions.  Going forward, we’d also like to support more client and server side platforms including javascript, php, .net and java.

Creating Server methods.

Delphi developers will most likely want to add their server methods to a TDataModule class.  Delphi classes that contain server methods must be compiled with METHOD_INFO enabled because we use
Delphi’s RTTI for the dynamic invocation of methods..  All server methods must be made public.  Here is a simple example of a TDataModule with server methods:

Server methods parameters directions can be specified as input, input/output (var), output (out), and return (assign to Result).  Parameter types supported include the scalar values supported by dbExpress/ADO.NET drivers such as integer, string, date, time, floating point, etc.  Very large binary parameters can be passed as a TStream.  A TParams object can be used to pass an array of values or to return a single row result to the client.  Table shaped objects such as a TDataSet or a TDBXReader can be used as parameters and return values.  On the .net platform, a DataReader, DataView or DataTable can be used as parameters by the client.  Note that for table parameters the object type used on the client does not have to match the object type used in the signature of the server method.  This is because the messaging layer maps all table objects to a TDBXReader.  Custom TDBXReaders are fairly simple to implement.  It should not be too difficult to map a collection of object for an OR mapping technology into a TDBXReader inside a server method and return it to a client which reads it into a TDataSet.The implementation of these simple server methods is listed below:

Exposing server methods to clients

Once you have implemented your server methods, they must be exposed by a DataSnap server.  This is easy.  Its best to create a separate DataModule for your DataSnap server.  Create a new DataModule and drop three components from the new “DataSnap Server” component category:  TDSServer, TDSTCPServerTransport, and TDSServerClass.  Set the TDSTCPServerTransport.Server property and the TDSServerClass.Server properties to the TDSServer.  The TDSServerClass.OnGetClass event must be set to your server class like this:

Now execute the application and the server will be started automatically.  There is a TDSServer.AutoStart property that defaults to true which causes the server to start when it is notified that the component is being loaded.

Calling server methods from a client

From the client’s perspective, server methods look a lot like stored procedures.  There are some notable exceptions though.  Server methods can send multiple table objects from the server to the client and from the client to the server.  Here is a simple example of a client calling one of the server methods above from within a dunit test case:

Using server connections

The DataSnap dbExpress connection can be used by a client application to access a server side dbExpress connection.  By default, a DataSnap dbExpress connection is only able to execute server methods on the DataSnap server it is connected to.  If the TDBXPropertyNames.ServerConnection property is set to a server method that returns an instance of TDBXConnection, then the client side DataSnap connection will also be able to execute sql commands against the returned TDBXConnection.   Here is an example of a server method that returns an instance of TDXConnection:

The goal with this example is to keep things short and simple.  Note that some
Delphi developers have pointed out that they prefer to use a separate try/finally for each object allocation with the allocation being made before the try.  An advantage of this approach over the one used above is that if an exception occurs in the destructor of an inner finally block, the out finally blocks will still be executed.  Another point that has been made is that the calls to FreeAndNil here are overkill since they are never accessed after they are freed.  As a matter of habit I like to use FreeAndNil in dunit test cases because for us there tends to be a higher likelihood of reallocating an object in a test case method than in a typical application method. A client side connection can establish associate itself with this connection by setting its ServerConnection connection property to:ServerConnection=ServerMethodsDataModule.GetServerConnection

Hosting existing remote data modules

RemoteDataModules are configured similar to any other server class.  As with other server classes, you will need to use a TDSServerClass component and wire the OnGetClass event to return your RemoteDataModule class.  On the client, you will need to use the new TDBXAppServerConnection.

Share This | Email this page to a friend

Simple deployment.

There are single file deployment options for both the .NET and Java platforms.  If you use the windows version, there is only one .NET assembly you need to deploy.    This is particularly nice for ASP.NET applications.  No longer need to rent a database from your ISP.  On the Java platform there is a single jar file to deploy.  The license file also needs to be deployed, but this is small and it is “not” machine locked.  This makes it easy to do simple xcopy deployments.  On both platforms the delployment size of the binaries is about 1.3 megabytes.

Industry compliance.

SQL-92 compliant.  Local and remote ADO.NET providers.  Local and remote JDBC drivers.  DBX4 100% Object Pascal win32/.NET driver.

Performance.

Early on we would do benchmarks against other Java databases.  However we quickly found that we were orders of magnitude faster than them.  We also found that we had no problem competing with databases written in native code.  Invariably, every time we found ourselves being beat it had nothing to do with managed code vs native code.  We could always match or beat the competition with better algorithms.  I know native code can be made faster for simple code, but with the complexity of transactional, high concurrency database kernel, better algorithms are king.  We have not performed major tuning for a while, and I’m not going to claim it’s the fastest, but this database is fast.

In-process execution.

The local ADO.NET and JDBC “local” drivers execute in process for improved performance and ease of deployment.

Delphi, C#,  VB.NET, and Java stored procedures and triggers.

These are safer and more powerful languages to write database stored procedures and triggers in.  In addition to using mainstream languages, you can also use off the shelf IDEs, debuggers and profilers to develop your stored procedures and triggers.

Really cross platform. 

Blackfish SQL pretty much runs every where .NET and Java run. We also have built a compact framework assembly for Blackfish SQL for the compact framework as well.  We are not ready to release this now, but we know it can be done. 

Unicode.

This is the only way Blackfish ever knew to handle character data.  Dealing with character data is always simple.  No multi-byte, national character, etc issues.  You also don’t need to specify the length of a string for performance reasons.  String storage is transparently managed.  For collation we use the collation key support provided by the .NET and Java platforms.  Why reinvent the wheel? 

64 bit support.

We didn’t have to lift a finger for this support.  If the .NET virtual machine is 64 bit, you can have a 64 bit database cache.  The same for a 64 bit Java vm.  Same binary for both 32 and 64 bit platforms.

Security.

Databases can be password protected and there is SQL based security including support for oles.  The database file itself can be encrypted using the Two fish 128 bit Two Fish block cipher.  Don’t forget your password.  There is no back door.  Some security conscious companies and government agencies view 100% managed code solutions as more secure.

BDE.

We have mentioned in our roadmap that Blackfish SQL as a BDE replacement.  Today this is still a bit of work because it requires that you move away from the simple TTable/Query navigational data access model.  Going forward we will be looking at what we can do to support a navigational model to facilitate BDE application migration to Blackfish SQL.  The idea is to support native win32/64 client applications that can connect to a Blackfish SQL server using a navigational data access model (TTable).

Tooling.

Our Blackfish SQL tooling is very basic.  Leonel put a lot of work into adding ADO.NET, dbExpress 4 and query builder support into the DataExplorer in a relatively short period of time.  Our plan is to improve the capabilities and extensibility of DataExplorer going forward not just for Blackfish SQL, but also for Interbase and the rest of the databases we support as well.  In the mean time, Blackfish SQL has built-in “DB_ADMIN” stored procedures that allow applications to perform many common administrative tasks including database backup/copy/defragment/verify/encrypt/configuration, change password, metadata retrieval, active connections, lock manager state, and server side logging configuration.  The built-in DB_ADMIN stored process allows us to avoid polluting the name space of our standards based SQL dialect with Blackfish SQL specific extensions.

Share This | Email this page to a friend

This 4000 lines of code also replaces the all of the BDP ADO.NET 1.1 driver.  The reason this is possible is that we pushed most of the capabilities needed for an ADO.NET 2.0 driver into the dbExpress 4.0 framework.  In this way our win32 customers can benefit from the same features.  The new dbExpress 4.0 metadata is a perfect example of this shared capability.

BDP was implemented in C# and directly implemented the deprecated ADO.NET 1.1 interfaces.  This prohibited win32 applications from benefiting from any BDP improvements over dbExpress.  ADO.NET 2.0 also implemented many BDP extensions in an incompatible fashion.  Supporting both dbExpress and BDP would have created a lot more work for our development team.  Its maddening to have to maintain two separate code bases that provide many of the same capabilities for 8 or so different database vendors.

Unlike BDP, the AdoDbxClient bridges from Ado to dbExpress.  This approach allows us to “adapt” ADO.NET 2.0 to dbExpress 4.  This also makes it easier for us to adapt to future changes in ADO.NET.  When you don’t control an API like ADO.NET, it is better to bridge than to extend.  This allows us to continue improving dbExpress and adapt to changes in ADO.NET as needed.

This is one more significant step towards making it easier to write database drivers using Delphi.  A conformant dbExpress 4 driver with dbExpress 4 metadata support can now be surfaced as an ADO.NET 2.0 provider with no additional work.

Our development team is very good at creating leverage.  With each new release we create new features along with new foundations for features in future releases.  The ADO.NET 2.0 provider is a perfect example of how this approach has paid off.  The dbExpress 4 framework was surfaced in our win32 release.  With the addition of broader and more consistent metadata, we were able to create an ADO.NET 2.0 provider for 9 different dbExpress drivers in a small number of months.

Share This | Email this page to a friend

In the effort to consolidate the dbExpress and BDP driver frameworks into one we needed to enhance the metadata for dbexpress.  The dbExpress 3 metadata is written in C.  The BDP metadata is written in C#.  The dbExpress metadata was not rich enough for the tooling infrastructure we want to support in the future for both win32 and .NET.  Our solution was to implement new metadata for all drivers in Object Pascal.  This can be compiled for both win32 and .NET.

On the .net side this is also the metadata we use for our ADO.NET 2.0 AdoDbxClient provider which allows any of our dbExpress drivers to be surfaced as an ADO.NET 2.0 provider.

Another interesting aspect is that the new metadata is provider based making it pluggable.  This allows other dbExpress driver implementations that use the same back end to reuse our metadata implementation.  It also allows us to potentially support other vendor’s ADO.NET drivers with our tooling.  For example, the DataExplorer can use Microsoft’s ADO.NET driver with our MS-SQL metadata provider.

The provider is specified in the dbxdrivers.ini.  If this entry does not exist in the dbxdrivers.ini file, the metadata request will be passed on to the underlying driver.  In the case of our dbxDrivers, you will get dbExpress 3 metadata support.

I’d say about 25% of the dbExpress driver implementations deal with metadata support.  There is already a large amount of framework implementation shared for all drivers.  With the new metadata we are one more significant step towards a 100% Object Pascal solution.  Object Pascal provides us with the portability we need to support win32, .NET and eventually win64.

The new metadata commands are documented in the DbxCommon.pas.  There are also several classes with constants and documentation for the new metadata tables and their columns.  The new metadata extends dbExpress 3 metadata with support for retrieving:  data types, views, synonyms, catalogs, schemas, procedure source, package source, package procedure parameters, roles, and reserved words.

We also now provide driver independent support for editing metadata.  Support for creating sql dialect sensitive create, alter, drop statements was initially only provided for tooling purposes in the DataExplorer.  However we have now also exposed a DbxMetaDataProvider class that surfaces this capability for applications.  This will make a slight increase in the size of your application since you will need to include the metadata writers.  The ability to generically create tables is useful for many applications.  The interface allows you to describe what a table and its columns should look like.  This description is passed to the TdbxMetaDataProvider.CreateTable.  Here is a very simple example that shows how to create a table with an int32 column named "C1", a decimal with a precision of 10 and scale of 2 named "C2" and a character based column with a precision of 32 named "C3". 

var
  MetaDataTable: TDBXMetaDataTable;
  DataGenerator: TDbxDataGenerator;
  Command: TDBXCommand;
  Row: Integer;
begin
  MetaDataTable := TDBXMetaDataTable.Create;

  MetaDataTable.TableName := ‘QCXXX_TABLE’;
  MetaDataTable.AddColumn(TDBXInt32Column.Create(’C1′));
  MetaDataTable.AddColumn(TDBXDecimalColumn.Create(’C2′, 10, 2));
  MetaDataTable.AddColumn(TDBXUnicodeCharColumn.Create(’C3′, 32));
  MetaDataProvider.CreateTable(MetaDataTable);

TDBXInt32Column, DecimalColumn and UnicodeCharColumn are pre fabricated class extensions of the TDBXMetaDataColumn class.  The TDBXMetaDataColumn class contains several column properties including data type, precision, scale, is fixed length, is Unicode, etc.  You can create your own pre fabricated classes for column types common to your application.  The MetaDataProvider will find the best possible match of each TDBXMetaDataColumn to the underlying type system of the targeted database backend.

Share This | Email this page to a friend

We’ve been going about 90 miles an hour this summer working to get the next release of BDS studio out.  On the database side of the house there are a ton of new features.  I don’t know what we were thinking putting so much into the product and shipping at the end of the summer.  The good news is that there’s lots of cool features to talk about now.  In this blog I’m going to talk about one of the big features – Blackfish SQL 8.0.

Blackfish SQL is an embeddable database which is now a feature of Delphi.  Delphi Pro, Enterprise and Architect customers have unlimited deployment licenses of Blackfish SQL.  There are file size and user limits, but these limits should be acceptable for a large number of smaller scale applications.  Deployment licenses can be purchased if the user or file size limits need to be increased.

The Blackfish SQL database was written completely in Java for its first 7 revisions.  The database product was called JDataStore then.  A couple years ago or so Jens Ole created a Java to C# translator so that we could create a .NET version of the database.  The bulk of the database kernel, query engine and driver layers are written in Java with minimal dependencies on runtime classes.  Since at its core the C# language and .NET runtime is semantically very similar to Java’s, and because we had minimal runtime dependencies, the Java to .NET translator came together fairly quickly (well, much faster than rewriting the database).  This translator allows us to continue extending the internals in Java and then automatically translate to C# to produce the .NET edition.  There is a very high degree of compatibility between the .NET and Java editions.  .NET clients can connect to Java servers and visa versa.  The file format is compatible as well.

We also have introduced a 100% Object Pascal DBX4 driver that can connect to Blackfish SQL called DbxClient.  Source code for this driver is included in the product.  It can be compiled for either win32 or .NET. Because it is 100% Object Pascal and requires no client library you can even build “big fat win32 exes” that connect to either a Blackfish SQL for windows (.NET) or Blackfish SQL for Java server.  No additional client library is required.  This is a perfect example of why I really like seeing database drivers implemented in Object Pascal.  This allows us to use the same code for both .NET and win32 as packages or linked directly into the exe.  When there is 64 bit Delphi we can reuse the same code yet again.  What other language and runtime can do this as elegantly as Delphi (or even at all)?  To me this is one of the many reasons Delphi just flat out rocks.

What else is cool about DbxClient?  Well, its not called the Blackfish SQL DBX4 driver.  It’s a driver that remotes the interfaces of the DBX4 framework.  It can connect to anything that understands its JSON based streaming protocol.  Today only the Blackfish SQL for windows and Blackfish SQL for Java support this protocol.  However, I’d really like to see if we can leverage this technology as a connectivity solution for DataSnap as well.  I view a database driver is a fairly capable RPC layer.  The streaming protocol is currently a hybrid where it will mix binary data with text data.  In the future we’d also like to implement a full text mode to support other transports like http(s).  The binary hybrid mode is more compact and performant.

Another interesting aspect of the DbxClient is the source code.  I already mentioned that it was 100% Object Pascal compliable for win32, .NET and someday the win64 platform.  If you look at the source code you will notice that some units are located under a pas directory tree and some are located under an xpas directory tree.  The source code in the pas directories was authored in Delphi.  The source code in the xpas was not.  Jens Ole has taught his Java to C# translator a new trick.  It now can also translate from Java to Object Pascal.  Its pretty hard to tell it came from Java though.  Typical Delphi naming conventions are applied as part of the translation process and multiple related classes are packaged into single Delphi units.  Often times a Java package of classes logically maps to a Delphi unit.  Why do we do this?  By using Java as “pseudo code”, we can maintain this JSON based streaming protocol on 3 different platforms:  Delphi/C++ win32/win64, .NET and Java.  Keep in mind we have a .NET and a Java server at the other end of the DbxClient that need to be able to interpret the same JSON streaming protocol.  This same technique was also applied for the new DBX4 metadata.

DBXClient also has some DBX4 extensions.  It is the first dbExpress driver we have created that supports streaming blobs and parameter metadata.  You don’t need to read the entire contents of the blob in one shot.  This is not that big of a deal for things like pictures.  But for things like video streams you really want to be able to stream the blob.  The DBXClient also supports parameter metadata.   If you don’t add parameters to the command, it will add them for you when you prepare the statement.  This saves coding and is less error prone than creating parameter objects yourself.  I’d like to make these important capabilities part of a “DBX5” in the future.

That’s enough for one blog…

Share This | Email this page to a friend

I was chatting with Dan Miser about the new Delphi 2007 yesterday and he made an interesting comment about the product:  “yeah. i like the renaissance going on with Delphi”.  I thought to myself that “renaissance” was a really cool way to describe what is going on with Delphi and CodeGear as a whole.  I’d like to take a few minutes to describe what I think is fueling this renaissance.

I think its interesting to take a look at what is going on in the inside of the Delphi team.  Many measures were taken during this development cycle to improve the quality of the product and the productivity of the team.  The product is continuously built and tested as developers check into the source code repository.  Its pretty much a criminal act to check in something that either breaks the build or our automated tests.  The obvious benefits of this are that you ship a better quality product in a timely fashion and the overall productivity of the team is increased because we don’t loose as much time to product breakages.  There were also several other key efforts to improve our automation of the mundane including string resourcing, help system production and the construction of developer builds.  In addition to process improvements the team as a whole really pulled together in these last two months working really hard to create the best possible Delphi 2007.

CodeGear Enablement.  When it comes to management, you really have to watch what is done and not what they said.  I do think many positive steps have been implemented to enable CodeGear products and their respective teams to succeed.  This isn’t rocket science, just good old fashioned common sense.  Enable CodeGear to focus on the tools business.  Pay attention to developers.  Continue to invest in successful products.  Invest in new products.  The morale and energy in CodeGear is much higher than it has been in years.  In Scotts Valley alone we have many talented employees who had previously left the company who have now returned.  We have many talented new employees as well.  A high quality Delphi 2007 release was delivered in a timely fashion.  We are on the verge of releasing several other products.  These are the positive signs and benefits of enablement.

I think what makes great products is the convergence of motivated and talented people who care about the products and customers.  To enable this convergence the products and teams must be provided the environment and means to succeed.  These things are all key to creating a renaissance.

-Steve

Share This | Email this page to a friend

In my first blog on the new DbExpress driver framework concerns were raised about its performance.  I worry about this a lot myself.  The initial goal was to make sure we didn’t make anything slower.  Justin is a member on our team who has completed some preliminary tests that show that there is no measurable degradation between dbExpress 3 and 4.  In some cases the performance has actually improved.  There are so many variables in performance analysis and tuning.  All of our tests are still preliminary and need more investigation.  So I am not going to that we are as good or better across the board.  At least not yet.

The dbExpress 4 vcl (the SqlExpr unit) is now single sourced and sits on top of the new dbExpress 4 driver framework.  This makes for a clean separation of component and driver layers.  The dbExpress 4 driver framework is also fairly easy to use, it just doesn’t have all of the features of the components that sit on top of it.

So what makes dbExpress 4 faster?  If you use the new dbExpress 4 driver framework directly there will be less over head because you are little closer to the metal.  Scans of large tables don’t usually show much of a win.  But there is a small, significant difference when executing multiple short duration select statements.  Execution of non-select statements is about 2 times faster on the native platform.  This is due to the fact that the vcl was closing the statement for every execution of non-select statements.  Turns out that there were 3 or 4 drivers that had bugs where they did not support executing a parameterized non-select statement more than once.  You never saw this problem because the vcl always closed the statement.  We fixed these problems for BDS2007.

On the managed size, the performance difference between BDS2006 and BDS2007 vcl is much larger.  I have not had time to get to the bottom of this, but one reason may be the way we use pinvoke in the new framework.  In the new framework we try to provide pinvoke with as much information as possible about what we are trying to do.  Instead of passing data by pointer, the values are typed.  For example, instead of passing a pointer to a buffer that contains an int, we pass an int.  Pinvoke can choose to just pass this on the stack.  For Strings we pass String, for Strings we want return values for we pass a StringBuffer.  Pinvoke knows how to handle both.  We also mark all parameters with in and out attributes as needed.  The thinking is that we will provide as much information to pinvoke as possible.  This will at least make the interaction more safe and will hopefully also provide better performance.

The new dbExpress 4 framework also has a new TDBXPool connection pool.  If your application frequently opens and closes connections, this will provide a significant performance boost.

The analysis is not complete, but I don’t think most existing applications will see any significant degradation.  If you use pooling or start using the dbExpress driver framework directly, you will see some improvements for connection opens and short duration statement executions.  The improvements on the managed size have many times better than for the native side.    

Share This | Email this page to a friend

Michael Swindell just came in my office and asked me to blog on what we are cooking up in the database side of the Delphi house.  I’m pretty busy right now, so I’m just going to rattle off some things that I hope developers will find interesting.  Excuse me if I ramble, I’m not going to spend a lot of time editing this.

I am not new to database technology, Borland or CodeGear, but I am relatively new to Delphi.  I joined the team last May.  Coming into this team I was fully aware of the proud Delphi heritage as tool for rapidly building database applications.  No question about it, this continues to be a key strength of the Delphi product.  However over time, several database related technologies have been introduced that either need modernization or provide redundant solutions to our customers.  Our customers need cutting edge technology, innovation, and consolidation of our technology offerings with as much backward compatibility and ease of migration as possible.  That’s a mouthful and an ambitious goal.  However we have a strong development team in place and with ongoing involvement of the Delphi developer community, I think we can pull it off.

Delphi the Super Platform

I think when the .net framework first came out, many people thought it would replace most interest in developing native applications.  This thinking proved to be wrong.  Hindsight is 20-20 though.  We have customers that develop both native and managed applications.  We do not want to treat either as second class.  Luckily Delphi is the only language and runtime that I know of that allows software to be developed for both native and managed applications in a “first class” fashion.  There are ways to compile both .net and java applications into native executables, but they are just bringing their managed runtime into the app as a separate subsystem.  To me this is not first class.  Delphi on the other hand is more of a proper managed app when executing under .net and a proper native app when executing under the native win32 platform.  For example, Delphi will use .net memory management when executing on the .net platform and Delphi will use the existing highly optimized native memory management when executing on the win32 platform.  To me this an important feature that makes Delphi a “super platform”.  Microsoft with enormous R&D resources could not achieve this kind of cross platform support for VB.  Delphi just turns out to be better suited as a super platform.

The new software we are writing is single sourced.  We compile the same source code for both win32 and .net.  We build and run our dunit test suites every day for both win32 and .net.  It works well and allows us to provide a “first class” feature set for both our win32 and .net customers.

Technology Consolidation

Delphi has some redundant database related technologies.  Sometimes we will try to consolidate these offerings.  This allows us to put more energy into a common solution and simplify the choices for our customers.

When I started last May we had two database SQL connectivity technologies:  dbExpress and BDP.  Delphi provides 8 dbExpress drivers and 8 or so BDP drivers out of the box.  The internals of the dbExpress and BDP actually share a common heritage.  They were both created by the same developers, but they are entirely separate code bases.  This is a lot of drivers to enhance and maintain.  Early on I made the call to consolidate these two technologies into one.  The result is the dbExpress 4 framework that will be included in the next Delphi result. The dbExpress framework is written in 100% Delphi.  This replaces the com based dbExpress 3 framework.  The dbExpress 4 framework is single sourced so applications that use the dbExpress framework can be compiled for both win32 and the .net platform without any source code modification.

So what about BDP? It provides our customers with an ADO.NET solution.  However BDP is based on an older “interface” based version of ADO.NET.  The newer ADO.NET 2.0 introduced an interface based on abstract base classes and deprecated the more brittle interface based solution.  In my opinion, implementations of the ADO.NET abstract classes should not add methods to their implementations to add features.  This is Microsoft’s name space.  However, I want us to be able to innovate in our connectivity solution.  This is why I decided that we should create a “bridge” from ADO.NET to our new dbExpress Drivers and not “extend” the ADO.NET classes.  This allows us to move forward on our dbExpress technology and use a very lightweight bridge to provide a single ADO.NET driver.  In many ways the dbExpress 4 driver framework is a superset of the ADO.NET technology.

The end result is that we have cut the drivers we must enhance and maintain in half, all driver features are provided to both native and managed applications, and we have an ADO.NET 2.0 driver that bridges to any dbExpress driver.    BDP and its support for the older version of ADO.NET 1.1 will be replaced with a new ADO to dbExpress bridge This bridge is implemented using the newer ADO.NET 2.0 abstract base classes.

Product Innovation

We have several innovative technologies in the works.  I’d like to talk about some interesting driver extensibility features that came out of our work to consolidate the dbExpress and BDP driver solutions.  The new dbExpress 4 framework introduces “delegate drivers”.  Delegate drivers are light weight and can be used to provide both pre and post processing of all public methods and properties of the dbExpress 4 framework.  There is a single connection property used to place a delegate driver in front of a regular dbExpress driver.  Here is a snippet below from my dbxconnections.ini file:

[IBCONNECTION]

DelegateConnection=DBXTraceConnection

DriverName=Interbase

Database=workerbee:c:\BorlandInterbaseexamplesdatabaseemployee.gdb

…

The first line indicates that when we connect to IBCCONNECTION, it will be delegated to by the DBXTraceConnection.  DBXTraceConnection has properties in the same dbxconnections.ini file.  So its treated very similar to a full fledged driver.  Earlier in the same dbxconnections.ini file there is an entry for DBXTraceConnection as well:

[DBXTraceConnection]

DriverName=DBXTrace

;TraceFile=c:\tempdbxtrace.txt

TraceFlags=PARAMETER;ERROR;EXECUTE;COMMAND;CONNECT;TRANSACT;BLOB;MISC;VENDOR;READER;DRIVER_LOAD;METADATA

;TraceDriver=true

;TraceFlags=NONE

You can also chain delegate drivers.  For example, you can have a connection pool delegate that delegates to a trace delegate which in turn delegates to an Interbase driver.  We will included full source code for two dbExpress delegate drivers:  TDBXTrace and TDBXPool.  TDBXTrace provides framework level tracing.  The output is kind interesting.  Instead of “binding” this or “describing” that (who understands that stuff any way?), it outputs Delphi code that indicates how the framework is being used.  Here is a snippet from the execution of one of our dunit tests:

{CONNECT        } Connection2.Open;

{COMMAND        } Command2_1 := Connection2.CreateCommand;

{COMMAND        } Command2_1.CommandType := Dbx.SQL;

{COMMAND        } Command2_1.Text := ‘SELECT * FROM BASICDML order by F_VARCHAR’;

{PREPARE        } Command2_1.Prepare;

{COMMAND        } Reader2_1_1 := Command2_1.ExecuteQuery;

{COMMAND        } Command2_2 := Connection2.CreateCommand;

{COMMAND        } Command2_2.CommandType := Dbx.MetaData;

{COMMAND        } Command2_2.Text := ‘GetIndexes "workerbee:c:\BorlandInterbaseexamplesdatabaseemployee.gdb"."sysdba"."BASICDML" ‘;

{COMMAND        } Reader2_2_1 := Command2_2.ExecuteQuery;

{COMMAND        } FreeAndNil(Command2_2);

So if you are a database developer most of this is going to be much easier to understand than typical database client library trace output.  It shows exactly what is being called in the dbExpress framework.  You can also enable lower level client library logging as well (this is what dbExpress 3 provided), but I have it disabled in this snippet.  The comment in the front of each line shows the trace category.  “Connection2” means that this was the second connection created since logging was started.  “Commnad2_1” means that this was created by connection 2 and that it was the first connection created by connection 2.  Reader has three numbers and as you can probably guessed, this indicates that it is the first reader created by the first command of the second connection.  This numbering scheme is especially useful for multi-threaded logging.

Take note of the line that sets Command2_2.Text to ‘GetIndexes’.  This demonstrates another innovative feature of dbExpress that I call “extended commands”.  Metadata requests pass parameters (i.e. catalog, schema, table pattern), and return a Reader.   Sounds like a command, so why not make them the same?   To support this, a command type property was added so that the driver knows this a metadata command, not a sql statement or stored procedure.  The command type is a string that enables us and other driver vendors to provide new name spaces for commands.

TDBXPool provides basic connection pooling for all dbExpress drivers.  This delegate driver also implemented its own extended command that returns a reader with a row for each connection pool that is currently active.

The combination of delegate drivers and extended commands makes for a very extensible driver framework.  Delegate drivers are trivial to write.  If you don’t like my connection pool or my tracing, it won’t be hard to replace them.  Or maybe you can improve them and give them back.

Ok, I better wrap things up now.  We have a lot more in the works for this year that I don’t want to talk about just yet.  Today I mostly touched on database connectivity.  However, this is not the only alligator we are wrestling with.  Tooling, runtime components, multi-tier development and core database technology itself are all in the works or at least on the radar.  We also fixed some really old bugs this year.  We have a great development and QA team skilled at creating highly leveraged solutions.  I have met many Delphi developers that are very intelligent, highly skilled, and passionate about Delphi.  Delphi developers also tend to be incredibly savvy about building database applications.  I am counting on the Delphi developer community to constantly be looking closely at we are doing and provide the kind of feedback that will make Delphi database technology the best it can be.  The end result will be much better if we have lots of bright minds critiquing and extending the technology.  To that end we will try to continue offering open solutions with source code included when ever possible.

-Steve

Share This | Email this page to a friend

I recently made a passing comment on our Delphi database field test forum about my recent experience of learning Delphi.  I was quickly asked to blog it.  So here are some quick notes on my experience learning Delphi.  A little background first though.  I started my software career writing in c and some assembly, using make and vi.  While at Borland I moved on to C++, Java with JBuilder and Eclipse and a little bit of c#.  In my 17 or so years at Borland I had never developed in Delphi.  For the past 10 years I have been doing most of my development in Java.  I started developing in Delphi last May. 

Learning Delphi has been an interesting experience for me.  Initially I thought the different syntax might be a bit of a hassle.  Its easy for a developer to unnecessarily get hung up on things like editor key bindings and programming language syntax.  However, Delphi syntax isn’t really a problem for me.  Sometimes its actually an improvement.  For example, the notion of a “Result” variable for return values can come in handy when writing function logic and the property syntax makes for elegant, easy to read code.  At the end of the day the more important issue to me is the ability to create object oriented abstractions.  Delphi’s support for single inheritance, delegates (events) and interfaces is powerful and straight forward.  Records (value types) can be especially convenient and efficient for non-gc based memory management. Delphi’s reference based objects with fields that are set to nil on creation are efficient and less error prone.

Packages, units, interface/implementation sections were a little different from what I was used to.  These features do help to provide good structure and encapsulation for Delphi applications.

I remember years ago that Anders Hjelsberg was giving me a demonstration of a new “override” feature for virtual methods in Delphi.  He explained it more than once I think because I was not showing enough excitement for the feature.  At the time I was working in Java so it wasn’t something I was going to get to use anyway.  Now that I am developing in Delphi, I really like this feature.  Without the override key word, implementing virtual methods through inheritance is more error prone.  Sometimes it’s the little things that make all the difference.

The Delphi compiler itself is lightning fast.  Sometimes I compile twice because I’m not sure I actually hit the compile key.

The Delphi naming conventions for classes, fields and parameters kind of make me laugh – in a good way.  I remember when I was working in the JBuilder team in the early days of Java that the Delphi guys would come over and say:   “You know, you really should put a “T” in front of your classes, and an “F” in front of you fields”.  Well that wasn’t really going to go over well with the rest of the Java world.  However now that I am working in Delphi, I never have to worry about how to differentiate that names for parameters, fields, classes, and objects instances.  Its simple, it works, and it is expected by the rest of the Delphi developers in the world.

Another aspect of Delphi that has been very interesting to me is the ability to write software that runs in both native and managed platforms.  I don’t know of any other development environment that comes close to supporting this as well as Delphi.  I like this idea of the delphi language/rtl/vcl being a "super platform" for win32/64, .net, .net cf.  Of course going forward Delphi could become even better at this.  I have had to factor out platform specific code into separate units and use some ifdefs.  “Single source” may or may not be a big benefit for the Delphi community at large.  However, I like leveraging this feature so that we can provide similar or identical features for both the native and managed platforms that we support.

There are things I see missing and some things I don’t like in Delphi.  However almost all of these are correctable and the team already has plans to address many of these in future releases.

Well I’ve only scratched the surface of Delphi.  Overall I have really enjoyed developing software in Delphi.  Its well suited to rapidly developing large scale, high performance, object oriented software.

-Steve

Share This | Email this page to a friend

I recently joined the Borland DevCo spin off (April 25, 2006).  I have spent most of my working career at Borland between February 8, 1988 to October 14, 2005.  In that time frame, I worked with many great teams and projects.  Most of my work has been some how related to database technology.  I’ve always had a fascination with time and space trade offs.  Database related problems are full of these kinds of trade offs, so I guess that is one reason why I am drawn to them.  My interest in optimization goes beyond the execution of software.  I also enjoy finding ways to create highly leveraged development teams through the development of reusable software, better tools and process.  Creating highly leveraged teams allows us to work in smaller groups, move quicker, produce more reliable software, and spend more time on the interesting problems.  As a developer this makes life a lot more fun.  Along these lines, I also have a keen interest in making sure our solutions are no harder than they need to be.

I have been hired on at DevCo in a leadership role for the Delphi database technologies.  I am still coming up to speed on my responsibilities and Highlander planning.  I already see some cool things we can do, but I’m still processing a lot of information and suggestions.

In my prior work at Borland I did not work directly as a member of the Delphi team.  We did collaborate on the shared designs between the JBuilder and Delphi from to time to time.  I have a very high regard for the Delphi product, team and the development community.  Although I have a lot of related experience in the context of other Borland products, I still want to develop a good understanding of what it means to be Delphi.

I thought it might be a little interesting to talk briefly about the new old guy’s first impressions of DevCo.

I really, really like the products.  They are a big part of the Borland heritage.  Going forward there is a lot we can do to grow the existing products and to introduce exciting new products.

There is a good talent base at DevCo.  We are a little too lean in a few places, but this is correctable.  I and others here at Borland have already been contacted by several people who would like to join DevCo.  Many of these people are “quality” ex-Borlanders.

DevCo has a solid revenue base from a collection of IDE and database products.  There is a real company here with real growth potential.  For many years a large part of the profits from these products have gone elsewhere.  The Borland management has been more focused in other markets for some time.  Reversing this trend will create growth for DevCo.  We have the opportunity now to focus and invest in our products and our customer base.

That’s all I have for now.  I’m very excited about my new team and the potential of DevCo.

-Steve

Share This | Email this page to a friend