This page is intended to provide more in-depth detail about my career for those who wish to know and attempt to answer the inevitable "...just exactly what did you do..." questions.

Philosophical Ramblings


During my career, a few approaches have emerged that I carry with me:
(In no apparent order)

  • Even the scut work has to get done. No task is "beneath" me - be it cleaning bathrooms, making coffee, or designing and coding simple algorithms.
  • If I am leading, then I try and make sure everyone is doing something that they want to do and enjoy, and take on the less glamorous tasks myself. If I am a team member, I try and anticipate what needs doing and find ways to get things accomplished.
  • Look for ways to say "yes". Be a good volunteer.
  • Be gentle with criticism. Everyone is trying to do the best they know, treat them that way.
  • Have a thick skin - don't take criticism personally.
  • Accept that other's product will not be "the way I would do it" - you can't do everything, so get over it.
  • Get started - you can't accomplish anything unless you begin.

ADIC/Quantum

ADIC (acquired by Quantum) hired me to improve their build automation process. The SCM group had a shell based automation capability in place but it was becoming difficult to maintain (it was 3000 lines).

ADIC's environment is somewhat unique. One of their "claims to fame" is support on a wide number of host operating systems (somewhere around 35). This, coupled with the number of components (around 15) and the facts that they had "grown by acquisition" and that they had a geographically distributed enterprise (personnel in Redmond, Irvine, Texas, Colorado, the Ukraine, Minnesota, etc.) meant that the build processes for the various components each had a slightly different "flavor".

The client's desire was to use Perl. I architected an object-oriented, xml-driven, multithreaded application. The application worked fine except that it was a memory hog (a common complaint with Perl). I convinced them to allow me to port the application to Java. The Java port works very well.

The client also desired to implement a "nightly build". I installed/configured CruiseControl for this purpose. The "out of the box" implementation for the MKS repository was not quite correct, so I modified it (CruiseControl is open source).

HSA/Charter Communications

The work I did for HSA (High Speed Access)/Charter Communications started when I was hired as a backup for an employee that went on maternity leave. What was suppose to be a 10 week engagement turned into a nearly 5 year contract.

HSA was in the business of providing customer care and provisioning for cable company oriented internet access. Charter eventually bought them out. I was hired to provide software configuration management (SCM) capabilities, in particular "buildmeister". HSA's SCM group already had a fairly robust process in place. They used Remedy to drive the process and had architected things in order to allow multiple concurrent projects to proceed in parallel, all using essentially the same codebase. They were using CVS as the backend repository.

In addition to learning and executing the build process(es) for various products/projects, I was tasked with helping to improve the processes. They had created a home-grown project management system that used MS Excel. Unfortunately the task ids were generated by hand, and sometimes there were duplicates (leading to confusion). I wrote a VisualBasic script that automatically generated task ids, thereby eliminating duplicates. Eventually, I created an online, cgi-based application that took the place of the Excel spreadsheet. The application utilized MySQL for the backend database and provided for multi-role based editing/deletion/creation of tasks. The resulting table had multiple levels of sorting and user selectable views, all implemented in ECMAScript (JavaScript).

The Remedy driven build process originally was a bit of a problem for the development team. The process required that the populate by hand a field in a Remedy "change request" for each change they checked into CVS. This was error prone and not appreciated by the staff. I wrote a Perl based program that used several of the CVS "hooks" to automatically populate the appropriate fields in Remedy (using ARSPerl, a Perl API for Remedy). This removed a fair amount of contention/friction between the SCM group and the development group.

The original build process was done by hand. I wrote a Perl cgi program (the "Release-O-Matic") that provided web based automation. This cut the time for a build from 3 days to 3 hours.

I "inherited" the various application(s) deployment and configuration processes. I modified the codebase so that configuration variables were "discoverable", removing a long standing problem with new configuration variables being introduced and not documented until Test (sometimes) discovered them. I added automation to construct Solaris packages for the deployment and configuration of the applications.

Eventually, Charter decided to use BEA Weblogic as the application server. I was tasked with installing, configuring and troubleshooting a multitiered, multi-server environment utilizing Apache (for frontend servers), BEA Weblogic (for the application layer), BEA Tuxedo (for the provisioning layer) and Oracle (for the database layer). Installation/deployment/troubleshooting of various web applications, sometimes requiring that the application be "exploded", configured, and then repackaged was required. The environments (there were three) were setup as fully redundant, fully failover capable, high availability systems. I inherited automated failover scripting where the backup server(s) monitored the primary server(s) and "took control" if things didn't look good.

Troubleshooter Network

The Troubleshooter Network has been made famous by its host, Tom Martino. At the time I got involved, Mr. Martino had just gone national and was beginning to host the radio show in various remote locales. He was using a custom designed application that utilized ColdFusion on the backend to communicate during the show with multiple producers (they answer the phones and screen calls). The application was having difficulty due to network latency issues.

The client wanted a fixed price contract to architect and implement an application that would provide for better producer/host interaction, and be configurable in such a way that they could offer use of the application on an application service provider basis. I provided a fixed price bid for 40 hours.

The architecture I came up with was essentially a "structured chat" application, with a producer applet, a host applet, and a fairly simple server on the backend (all implemented in Java). The applet's were restricted to using AWT. All elements of the applets (number of phone lines, colors, identifying labels, etc.) were configured using a property file.

I delivered the application on time, on budget (with a few additional features that improved the useability). The only issue that surfaced during operation had to do with network reliability. I implemented the server such that when the last producer/host applet disconnected from the "room", the server de-allocated all memory associated with the session. Unfortunately, due to network issues, all clients sometimes were "disconnected" simultaneously. The applets had been implemented so that they automatically reconnected, but the data associated with the session had been de-allocated. A quick solution was to startup a browser and open up a host or producer session on the same machine where the server was running.

@Hospital

Edenet is in the business of providing "the internet experience" to hospital rooms. Their current product is based on a set of computers located within the hospital that the screen output is modulated onto a TV channel and then displayed by a patient's TV tuned to that particular channel. This approach suffers from scalability problems, maintenance, and a few others. They want to change over to a STB (Set Top Box) approach where the STB is a computer running a variant of Linux and a resident browser. In addition, they wanted to re-architect the way they were delivering their content.

The STB browser is a proprietary browser, somewhat HTML 3.2 compliant. The browser and the fact that the display is a TV added a technical "twist" (kind of like using stone knives and bear skins).

The decision was made to use Jakarta Tomcat and JSPs for templates, and CVS for code/jsp/document management.

  • I set up the CVS repository,
  • ported over some reusable code from a prior project,
  • figured out how to configure/use Tomcat as a standalone "personal" webserver/jsp engine,
  • wrote (or modified) basic build scripts,
  • and wrote the first example jsps.
This way each of the team members could develop in their own "sandbox" on a shared Solaris/SUN system. I documented this in a "Getting Started" document that allowed new team members to get up and running quickly and with little support.

Based on Edenet's requirements, I architected and implemented a content access mechanism that allowed for language specific and hospital specific content to be served. I helped select a content management system (CMS) vendor - eBt's Engenda/Dynabase and integrated it into the already existing architecture and implementation. I determined how to dynamically generate and serve Video On Demand (VOD) information (via a custom Servlet).

Kiva Genetics/DNA Sciences


By the time I became involved in DNA Sciences (Kiva Genetics) development effort, Vignette's Story Server had been selected as the server/content management/template engine solution. Story Server's template engine is Tcl based - when I first started on the project I knew very little about Tcl. However, I rapidly learned what was needed (this is both a testament to how easy Tcl is to learn and my own learning abilities). The project team had decided not to use Story Server's content storage system, opting instead to use an Oracle based back end and incorporating TclBlend and Java classes to query the database. My initial assignment was to construct a questionnaire engine that provided questions and retrieved/stored answers via HTML forms. I also inherited the various user login processes (there were/are three).

During development, the project team was faced with a technical/political crisis - the main content provider was using XML/XSL and IE5 as a authoring/display environment. This allowed him to have a desktop "template engine". Because our content back end dealt fairly well with XML, we figured we could easily import his XML data and figure out how to display it via Tcl templates. We had not anticipated on how complicated his templates would be, or on his extensive use of Javascript. The political element involved dealing with the egg on our face for not anticipating this requirement. The technical problem was "how are we going to deal with this content in a timely manner" - we knew we could do what we had planned but that reverse engineering the XSL templates would be time consuming, error prone and we would have to repeat the effort whenever the content provider modified his XSL templates. This situation came up during a regular team conference call. I had been faced with a similar situation previously (where the content was not quite compatible with the means in which we served it, but was in HTML) and wondered if there was a way to easily convert XML/XSL to HTML. During the conference call I searched the 'Net for XSL to HTML converters, and found xalan - a free, Java based XML/XSL translator. I downloaded the code and started learning how to use it. By the end of the day I was able to convert the provided content to HTML - however, the XSL syntax that IE5 expected was incompatible with the XSL syntax that xalan expected (but was close). In addition, there were various pathing issues, particularly with images and links within the site, where the XSL had to change. So that we would not have two different XSL files (leading to problems with maintenance) I invented a Java preprocessor and syntax that allowed the content provider to continue to use IE5 (and its XSL syntax) and when he submitted his XML/XSL, the preprocessor, acting on the syntax that I had invented, modified the XSL prior to using xalan to convert the XML/XSL to HTML. I wrote Java to post parse the resulting HTML and store it in our back end Oracle database. Tcl templates that I generated then retrieved the content and rendered it on the site. There were various ongoing issues that had to be resolved - generally with Javascript (since DNA Sciences wanted the site to be both IE and NN compatible, and the content provider could only test his Javascript with IE).

The team conference call was on Tuesday, and on Friday of the same week I was at the client demonstrating the capability, explaining the technique and negotiating with the content provider the details of the preprocessor syntax. This implementation was successful enough that the client subsequently expanded the use of XML/XSL and the conversion/parsing technique to "customized" markup language.

WebMD


WebMD was my first consulting as an "independent" contractor - I was hired on the recommendation of two former colleagues, practically without an interview. My first day I had to appear at the client's office in San Francisco, at the Presidio. I was met by one of them, Dr. Wes Munsil. Wes provided me with a "quick and dirty" bit of training, and, as soon as I had a network login, was handed my first assignment which involved writing templates to render content that had already been stored in a DB2 database (with access through Java). The template engine was WebMD's own, with its own syntax. By the end of the day I had checked in templates/code that accomplished what needed to be done (I believe I still hold the record for "quickest to contribute"). During the course of development I inherited various responsibilities:

  • Implementation of the "Organizer"

  • - the organizer (now MyHealthRecord) was a means for users to bookmark and organize various WebMD articles. It required the user to register (which I had to implement) and provided a means of capturing and storing information for a particular user. Prior to this, the entire site was user neutral. My initial cut at this was incorrect - I did not understand how the webserver (which was WebMDs own) dealt with templates and had used member variables incorrectly. A quick code review uncovered this mistake, which I quickly corrected. However, a bug uncovered during load testing kept us from deploying the organizer at first launch (the bug was subsequently fixed and the organizer deployed).
  • Integration of the "Health-e-meters"

  • - The initial launch of WebMD incorporated a number of interactive calculators, implemented in Javascript/HTML by another group of creative folk and programmers. It was my task to take what they provided and morph it into a form suitable to be served from WebMDs webserver. I also was tasked with discovering Javascript incompatibilities (we were suppose to support fairly old browsers) and in many cases fixing them.
  • Import of various articles

  • - the process of adding content to the database for later retrieval and display was termed "slurping". Each content source typically had different formats, information, etc. that had to have code written to modify/parse/format into a structure that could be stored in WebMDs database.
All of the above was for WebMD's consumer site. I also did some work on WebMD's professional site - maintenance for the physician portal, porting of the physician portal to WebMD's latest webserver, and construction of the nurse portal (which was never deployed). One of the more interesting challenges for the nurse portal was the importing of various full text and graphic content of books. Most of the initial content was somewhat HTML formatted, just needing massaging by an editor and then parsing/import into the database. One text, the "Nurses Physician's Desk Reference" was a different matter. It came to me on a ZIP disk as a Microsoft Access database. There was no description and the people responsible for the data and database were no longer available. I had to purchase Microsoft Access, then learn how to use it to export the data to ASCII files. Depending on the chapter, the database contained text interspersed with quasi-XML tags. There were no links for related terms and the tags were not consistently formatted. I wrote a translation framework in Java that read in the exported ASCII text files, "walked" through the input looking for strings that matched known tags/text, then passed the stream to an object whose responsibility was to generate correct HTML output. The framework was flexible enough so that we could evolve the translation and, if need be, only reuse the translator classes that made sense for a particular chapter/section. The resulting HTML was parsed and committed to the database in the same manner as the other books. The code was easy enough to understand and use that another programmer, Kurt DeShazer was able to help with creating various translations with very little help from me, and he used the framework for other translation tasks.

USWEST


The group at USWEST that I worked for had implemented and deployed a web application that performed address lookup and telephone number allocations for CSRs (Customer Service Representatives). The application accessed various legacy systems and presented a fairly unified user interface. However, USWEST was in the process of deploying a "single login" solution for access to legacy systems. The single login solution utilized an NT resident service (on the user's workstation) that accessed a database of usernames/passwords and provided them to the various NT resident applications that accessed legacy systems. They had not anticipated the need for web applications to need to do the same. My assignment was to suggest solutions, investigate their pros and cons, recommend a course of action, and implement a solution. The only documentation I received was the URL of the group implementing the single login and an overview of the codebase that implemented the web application (it was cgi-bin, written in C++).

Shortly after starting, USWEST telecommunications workers went on strike. All of the non-union employees were deployed to various places (most out of state) to "man the phones".

In spite of this, I managed to contact the program manager for the single login solution, get enough documentation that I could understand what needed doing, wrote a proposal and received approval, established a Java development environment. I implemented a no user interface applet that connected to the user's local workstation and queried for the needed username and password. To do this I had to use the Netscape Security API, which required that the applet be digitally signed. USWEST had not acquired a signing certificate and was in the process of deciding the companies direction with respect to digital signatures. So that we could proceed with development (and, as it turned out, deployment) I acquired a personal signing certificate. I also ended up providing support for other groups that needed to access the single login service.

Once the applet was deployed, I provided support and troubleshooting in situations where the single login failed (usually because the single login database did not have an entry for the specific user) and was assigned to implement a report generator (cgi-bin written in C++). Because USWEST's officially supported browser was Netscape Communicator 4.0 and above I had the luxury of implementing the web interface using Netscape's DHTML (Dynamic HTML). I was able to create a web application that performed like a resident application, with buttons that were grayed out until form input was correct, Javascript that performed input validation, etc. For the server side cgi-bin I implemented a reusable SGML library that minimized run time errors - in most cases, if the cgi-bin compiled the resulting HTML was correct. This application was used in its alpha state with no documentation or training.

TCI/CSG Systems


TCI was my first assignment as a professional programmer. The work was varied and most was done in C++ - later during this assignment is when I cut my teeth on Java. I wrote code at all levels - from library classes/algorithms that were reused by many applications to top level application code. I was a unique resource in that when I encountered a bug in an application I would fix the root cause, regardless of where it occurred. Some highlights:

  • TCI had decided to use Neuron Data's Open Interface widget tool kit, mainly due to multiplatform support. We (Scott Stodghill and I) extended Neuron Data's widgets by adding "data aware" behavior - data widgets, given basic information, "knew" how to access the enterprise Sybase database. The widgets were responsible for reading, formatting, providing edit capabilities, and mechanisms to commit changes to the database. I was responsible for these widgets and extending Neuron Data's graphical editor to support them. One widget of particular interest was the "Data LBox". Neuron Data provided a LBox (list box) that I extended to be data aware (now it could read and display row/column data). Neuron Data's LBox provided for in cell editing as long as the data was text - I extended it so that any other widget could be used as an in cell editor - Checkbox, Radio buttons, etc. This widget made possible applications that were pretty much just a single Data LBox!
  • I authored the VCM (Version Control Manager) application. This simple application had the following features:
    1. Was a single point of entry for any user of any of our applications. The user logged in, and once authenticated was presented a list of applications that were available (based on database information about the user's role/permissions).
    2. Insured that resident applications were the correct version, complete and installed. When a user "launched" an application, VCM checked to see that all of the required files were available, in the correct place, and that any installation scripts had been run. If a file was missing, VCM downloaded it from the database. Use of VCM minimized the amount of administration that had to be performed to distribute code, because it was a "pull" mechanism (vs. other techniques that involve "pushing" code onto workstations).
    3. Provided single logon. VCM communicated the user's username and password by writing them to a file (along with a time stamp) which the launched application read and then removed (I used a file due to multiplatform support issues). Because of security concerns, I implemented the DES (Data Encryption Standard) algorithm to encrypt/decrypt the information written into the file.
Development of this simple application took several technical turns - such as when we discovered that downloading the application when the user launched the application could take a fair amount of time (so I modified VCM so that it could run faceless as a cron job, looking for "pending" applications). VCM worked well when there was a large bandwidth pipe from the workstation to the WAN. However, there were a fair number of offices that were using fractional T1 (56K) to access the WAN. In such a case, VCM could easily saturate the pipeline, preventing users from performing their jobs. Even in situations where a large pipe was used, if several users needed an update simultaneously VCM could saturate the pipe because each workstation individually connected over the WAN. We came up with several different strategies for dealing with this issue, but finally settled on a revamp of the way that applications were distributed and installed.

Conceptually, we modified VCM so that when a file was needed on a workstation, the workstation "queried" other workstations on its subnet to see if any of them had the file. If so, then the needed file was acquired from that workstation. We used UDP broadcasts to "query" - that way, we retained the "administration free" capabilities of VCM. Because Java had built in UDP support (across multiple platforms) we decided to port VCM to Java. This marked the beginning of my experience with Java.

Storage Technology

Test Equipment Development

I was Senior Engineer, leading the development of a unit test station for STK's Iceberg DASD storage products. Each station was a PC running SCO UNIX (we later moved to a SUN platform) with various instruments contained in a VXI mainframe (from Hewlett Packard), another PC containing a set of IBM mainframe channel emulators and power distribution and control equipment. Hardware equipment cost was approximately $170K for each station and we built 28 stations. My team consisted of myself and between 3 and 5 software engineers, depending on workload/availablity. Total project budget was approximately $5M.

We were also tasked with creating reusable/reconfigurable components (both hardware and software) to support current and future test equipment projects.

This is where I cut my teeth on C++. Suffice it to say that we managed to make many of the classic mistakes in writing object oriented software using C++.

One other development of note involved a problem described to me by a colleague, Jim Adams. Jim was the engineer that created a PC hosted mainframe channel emulator/tester. He had a requirement to capture asynchronous events and an associated time stamp. The problem was that digital signals, when changing states from a 1 to a 0 or visa versa go through a "meta-state" where if you attempt to capture the state you may get the wrong state. This leads to an unbounded error in the value of the time stamp. Trying to design circuitry to avoid capturing the time stamp clock during a transition is difficult and has other tradeoffs. Jim was looking for a creative solution - I suggested that he implement the time stamp clock as a gray scale counter. With a gray scale counter only one bit changes between transitions, so the most that the time stamp could vary due to capture during the counter's metastate transitions was one bit (an acceptable error bound). That solved the metastate capture problem, but posed another - how do you convert gray scale back to binary? Jim could easily implement the gray scale counter, but we did not know the algorithm for converting back to binary (which is necessary in order to make sense of the captured data). I left Jim's office and described the problem to my team. Another colleague, Bob Bottlesen, puzzled out an algorithm by the next day! Jim and Bob did some quick tests that convinced them that the algorithm was correct - however, they couldn't be 100% positive because the desired counter was 64 bits (and even a computer program would take too long). I was able to prove (mathematically) that the algorithm was correct. Bob received permission from our manager to submit the design to EDN and they published it.

Systems Reliability Engineering

I worked for the Systems Reliability Engineering group first designing, constructing, executing and analyzing experiments do determine reliability figures of electromechanical parts/devices used (or under consideration for use) in STK products and then performing reliability analysis and prediction for new products.

Ball

Electronic Systems Division

DATATRACE is a miniature, environmentally sealed data logger. Its initial application was in the food industry, where after food is canned it is subjected to a process called "retorting" that sterilizes the can and kills whatever bacteria may be present in the contained food. There has been quite a bit of research into effective time/temperature profiles, but the research is generally based on "the point of least heating" which is roughly in the geometric center of the package. In practice, canners were unable to make measurements to insure that their processes were sufficient so they made sure that they had a large margin above the laboratory determined curves. This led to less than desirable results for the packaged food. DATATRACE was designed to be programmed, placed within the can, then recovered and read out yielding the desired time/temperature curve.

I was responsible for all research, design, firmware and production/test equipment for DATATRACE. I took the product from conception to completion - DATATRACE generated over $1M in sales the first year and continues to be a staple product for the company that Ball sold it to, Mesa Labs.

Computer Products Division

I started at Ball Computer products as a electronic technician, repairing circuit boards that went in Ball's OEM removable pack disk drives (back then, state of the art was 100 Mb, 5 platter, 14 inch packs!). I progressed through the ranks as I went to school at night. I started out as a electrical engineering candidate, but switched majors when I decided that engineering was mostly applied mathematics. Besides fixing printed circuit boards, I designed and built test equipment, troubleshot subtle problems, redesigned circuits to reduce their cost and improve their manufacturability, designed a new drive interface to meet specifications needed by Honeywell's Process Management Division, and many other tasks.

Contract Manufacturing Division

I began my adult career four days after I graduated from high school, working for Ball's Contract Manufacturing Division. I worked inspecting parts that were destined for IBM's copier production plant.