History

The concept for this software started back in the early 1990's in that I have always found that every application software development model and tool has been frustrating to use, awkward, and requires writing too much code that ended up being difficult to change when change was needed.  Most of the tools did not really address the needs of complex application development. New developments came along in Object-Oriented programming and Client-Server systems, Relational databases and languages like Java that all promised to make things easier by letting us create and use reusable objects.  Unfortunately, none of these ended up really delivering on the rapid application development promise.  All the 'add-on' technologies to simplify things ended up being more complicated than the problems they solved.

In the late 90's, I had the opportunity to lead a 5 person project team to develop some enterprise business software for a pipeline company, the software was very complex and we were building it using Java/Swing over Oracle.  At first we tried using early J2EE stuff to generate objects from entity definitions but we were quickly drowning in about 1.4 million lines of code that did nothing but need to be maintained as entities changed - and it was horribly slow (2 min to populate a drop down list with 2000 items) - we ended up throwing it all out and building a direct SQL/JDBC coded system that ended up performing very well (0.5 sec to populate the dropdown - and we cached it in memory so it took no time on subsequent uses).  One part of this system that allowed us to deliver solid solutions for customers was that we had a special table editor that would let you edit the configuration tables of the system easily, it was dynamic enough that if we set up the table in the right way, this editor would automatically pick it up an allow you to edit it like a spreadsheet.  Unfortunately, it was limited to handling single tables and relationships were all handled through drop down lists.  What we discovered was that 80% of the system would really be collecting and setting the data and the rules to govern the system as simple configuration tables and then we could focus on the real processes that needed to be coded into the system.  The software ended up being reasonably successful (in its niche market) and we developed it further into a Web based Java servlet application (Table editor was still in Swing) and adding many new modules for different customers even in multiple languages - all delivered successfully.  The key to the success we had in delivery was not to build exactly what the customer wants, but to build something that can be configured into what they are describing.  Take the requirements as the theme, build to the theme and make adjustments at implementation (subsequent implementations became even easier as the theme was already there and mostly required configuration and maybe new features added).  If you build the theme flexible enough, it can adapt as requirements change.  For example, when customers needed different allocation methods applied to some data, we didn't build actual allocation methods, we added a Jython/Python interpreter to allow us to plug in the essentials of the allocation formula as Python code and the database configuration would determine where each formula would apply.  This formula concept became used for all type of decision points in the system allowing logic to be used against data to be stored in the database with the data - it was very powerful - we could add new business logic in a very short time and apply through configuration.  Unfortunately, the business area didn't have enough business to really grow the concept but I could see that there was a lot of promise in much of what we did in terms of creating a framework for the application modules (like apps).  After leaving the company to pursue other shiny things, I parked the concepts I had learned but realized that they could be applied to any type of application if the right framework could be found.. or if needed, built.

Ten years go by and the application development landscape has changed dramatically except for the fact that many software project continue to run over budget and schedule (not mine of source) and are not very reliable or barely relevant in the end.  The tech landscape is now a huge mess of libraries, databases, web servers, web services, layers for communication, layers for object relational mapping, layers for UI, XML, HTML, CSS, JavaScript, etc.....  There are all sorts of tools out there that handle each of these concerns for you but by the time you get all these pieces working together, you will have no budget or time left to build the actually application you needed - and good luck keeping it all up to date.  

In 2009, I embarked on a pet project to tie all the actually essential pieces together into a frame work that would let you harness the power of these technologies without the horrible complexity because at their core, HTML, CSS and HTTP are actually pretty simple but powerful concepts and if each was only used for what it was intended but meshed appropriately with the other parts, it wouldn't need to be so complex.  This was when I came up with the concept and name PermeAgility - not just agile development, but systems and a framework that stays agile with the business.  I started with many of the concepts from earlier work (functions to generate html from java - see the Weblet class) but ended up reworking things many times along the way.  I discovered that we spend a lot of time coding interfaces and trying to get other peoples code to do something that we actually could have done easily ourselves, and the problem is we end up adding dependencies until dependency management becomes a real issue and system stability inevitably suffers.  In previous frameworks I have used, a simple hello world seems pretty simple, but its a long long way to a finished application with persistence, security, backup and recovery process, internationalization, etc...  In most frameworks, you have to add these aspects to your software, In PermeAgility, it comes baked into a tightly integrated core set of simple but powerful classes that each handle an aspect of the entire functionality in concert with the others.  The are no classes that just represent structure, there are no classes that are all useless getter and setter functions and a few overrides scattered about, there are no interfaces or implementations and definition classes, there are no classes that contain only a few lines of actual logic - each class does something useful and as independently as possible.

The first version of this framework was built on the H2 database original built by Thomas Mueller (also the developer of Hypersonic SQL).  It's a great database except that like all relational databases, it tries to be like other databases.  All relational databases generally support the same concepts across the board; tables and columns, keys, indexes and foreign keys except that you have to maintain all of the relationships between tables yourself and every time you need to get some data, you have to re-connect the data again using joins in every piece of SQL you write.  PermeAgility for H2 worked pretty well with but had limited functionality in terms of how you needed to model data to make the system work - I wanted something more flexible but all SQL databases support the same primitive values and awkward relationship handling and a few custom add-on capabilities that each database vendor adds to make themselves unique and not actually compliant with a standard that would allow for true database portability.  I lost interest for a while but continued to play with it once in a while as a way to practice and collect techniques and methods in one place so that I could have good examples for future work projects. JQuery was added sometime in this time frame.

In fall 2012, I picked it up again, and added more features and set up this web site for a place to collect documentation.  The initial documentation was done as part of a project in a COMP601 (research method) course I was taking at the time.

In Spring 2013, I became interested in NoSQL databases, I was taking the COMP602 (databases) course and I had realized that the problems inherent in standard SQL databases would not go away and that the SQL standard is an evolutionary dead end.  SQL databases will never scale well, they will always become too complex to handle as you add more tables and dimensions to the model and your SQL will get uglier and uglier and more difficult to maintain (possibly even before you get to implement if you don't change the model along the way to fit changing requirements)  NoSQL databases had emerged as an alternative to standard SQL, except that a new standard did not emerge, many different types of persistent data stores appeared, each with different strengths, different API's, different query languages, some distributed or clustered, some transactional, some non-transactional and non-concurrent.  Each database had different weaknesses as well, complexity of setup, deployment, maintenance, backup/restore, handling of relationships, indexes, and performance (usually a strength).  I wanted a scalable database that could handle a highly complex high dimensional model that was still ACID compliant and supported transactions.  Handling of primary keys and relationships was also very important, in the H2 version of PermeAgility, I automatically added a hidden integer primary to every table in the system, that way the system could automatically handle relationships but the problem was that many-many relationships were always awkward to manage and represent dynamically in a consistent way that made sense to the user.  I had a look at MongoDB, Redis, Cassandra, Neo4J and OrientDB along with others.  I initially really liked the performance, openness (open source) and scalability of the Cassandra database and started on PermeAgility for Cassandra.  A few weeks into the development, I realized that Cassandra was way too primitive for what I wanted to do and I wanted something that dealt with relationships in a better way with less work on the programmer's part.  Graph databases like Neo4J look interesting but they tend to be more for analytics than operational business systems.  Although OrientDB is marketed as a graph database like Neo4J, it also operates in a hybrid document mode which gives you the straightforward concept of tables (classes in OrientDB) along with the powerful link concept of a graph database - this was what I was looking for, a database where the data is essentially stored pre-joined but with much of the same flexibility as any other SQL database and ACID compliance to boot.  In OrientDB, there is no deciding a primary key, it is built in and handled.  You can make a single link from one record to a record in another table, or you can have table columns that have sets of links in ordered, unordered and mapped lists (mapped allows a name to be specified for each link) and navigating these links is fast so joins are rendered unnecessary.  It also supports vertical (table) and horizontal (row) security as a plus. This was the database where I could realize my vision of a simple framework over a capable database that links persistence, security, ui, and presentation in a way that allows application modules to be easily added to a solid simple foundation.  Adding AngularJS to the UI end also made the creation of components to handle link sets amazingly simple with very few lines of code. While this has not been without difficulty, I am actually surprised with how well this works and how easy the code looks now (it didn't look that way at first).  I still smile whenever I navigate to a record and see data from a perspective I hadn't expected shown automatically in a way that actually makes sense from that perspective.  

Right now, PermeAgility is a great playground to prototype and explore data structures and data security models of any kind and write simple powerful programs to do anything you want with that data without having to write a bunch of fluff code.  I plan to add lots of modules to support connections to other databases, file loaders (spreadsheets, csv, etc...),  modules to collect data from and control realtime internet devices, modules to support a more active graphical UI (I am calling it Visuility), a home automation and data collection module, a massive-multiplayer online game server and lots of actual business applications with this framework.  While development on the core will continue for purposes of hardening and performance, it is considered capability complete, future development will focus more on the plus framework and plus modules to allow the core to remain small and tight and support add on (plus) capabilities and plug in applications as needed for each unique implementation and deployment.  

If you are tired of XML configuration files, object-relational mapping, spring, swing, WSDL's, struts, design patterns, complex builds, difficult deployments, and nightmarish troubleshooting sessions, come to PermeAgility, I removed all that crap so you could just get stuff done.  If you find it useful for your purposes, I would love to hear about it.

Glenn Irwin - Dec 31, 2014  version 0.2.0