Reload That Config File

It is common for applications to load a configuration file on startup to control various options. Some applications can also reload their configuration file while running, allowing you to modify the application configuration without having to restart the application.

Contents

• Goals
• Dependency Injection
• Config Contents
• Config Objects
• Seven Steps
• Unit Testing
• Implementation Options

Goals

Configuration files (or config files) are useful because they let us change the behavior of an application with a mechanism that is much simpler and faster than modifying the application source and recompiling it. Being able to reload the configuration of a running application allows us to take that concept a bit further, as generally we can make reloading the configuration operationally simpler and faster than shutting down and restarting the application.

When reloading the configuration, we have the following goals:

• Reloading a configuration should be a simple operation for the operator to trigger.
• It should not be possible to load an invalid configuration. If the operator tries to do so, the application should continue running with the old configuration.
• When reloading a configuration, the application should smoothly switch from the old configuration to the new configuration, ensuring that it is always operating with a consistent configuration. More precisely, an operational sequence that requires a consistent set of configuration parameters for the entire sequence should complete its sequence with the same set of configuration parameters as were active when the sequence started.
• The application should provide feedback so that the operator knows what the application is doing. Logging, notification or statistics about configuration reloads should be available.

Dependency Injection

Dependency injection (DI) is a form of structural configuration in which different suppliers of a service are wired into an application based on the contents of a configuration file. In the typical case, these configurations are unlikely to change once an application has started. Although in principle it is possible to reload a DI configuration, and thus all of the discussion below could apply, in practice you might want to separate out the kind of relatively static structural configuration that is typically done with DI from the more dynamic parametric configuration that you might want to change while the application is running, and use different mechanisms to implement those two sets of configurations.

Alternatively, you can selectively disallow (as part of your validation step) configuration changes that are too much work to implement, requiring the user who wants to make such changes to restart the application.

Config Contents

If you think of a config file as being a set of late-binding commands for controlling the behavior of an application program, it should be clear that the most flexible config file is one that is itself a program. Applications that already have a built-in interpreter, such as emacs and applications written in Lisp, often simply feed their config files to their interpreter, giving them the full power of a Turing-complete language in which to express site-specific program behavior.

If you have an interpreter available, this can be a reasonable option: it is simple to implement, takes little work to document (assuming you already have to provide documentation for the interpreted language anyway), and provides a great deal of flexibility. One potential downside is that you might not want all of the power of the language to be available in a config file; in particular, if you are using the language internally, your program may have made available certain functions that you don't want a user to call from a config file. If your application already has a security framework built in to it, this may be easy enough to do, or you may not be concerned about it. In any case, you should at least be aware of this potential pitfall if you choose to use a language as your config file syntax.

At the other end of the spectrum, you could choose a standard name=value format, such as Windows INI file or a Java Properties file If you have a relatively simple application with just a few config parameters to set, this is probably a reasonable option.

You can treat all of your config data as strings and let the application deal with each individually, or you can define a set of datatypes that can be uniformly represented in a config file. This might include lists of data or other compound types.

One of the typical capabilities implemented in config systems is the ability to group config parameters into logical groupings. The standard Windows INI file does this with its [section] prefixes. You can simulate this in a Properties file by selecting a character to be a name separator (typically a period), then using that separator character to define names for your parameters that indicate their grouping. This can easily be extended to multiple levels to allow a hierarchy of grouped parameters.

Once you have groups of config parameters, you might want to implement some kind of inheritance mechanism, whereby you can declare a set of names and values in group A, then declare that group B has the same item values as group A, possibly with some specified exceptions. Or perhaps you would like to be able to set the value of a parameter to be the same as the value of some other parameter, or some combination or transformation of other parameters.

You can continue to add more capabilities to your config file, but once you start getting too complex, you probably want to adopt an existing language syntax to avoid creating something that is complicated to implement and maintain, tedious to document, and difficult to learn and use.

If you do use a language for your config file, you may need to modify your approach in order to be able to implement all of the steps given below. In particular, you should not directly modify your operational objects from the config file, as this violates the separation of config data from the application and makes it more difficult to validate the entire config before activating it. One solution is to make your config file code only set data into the new Config objects that are being created for the reload process. Other solutions are possible, such as setting up a mock execution environment in which code can be validated before being applied, but a detailed discussion of such techniques is outside the scope of this post.

When choosing a format, you might consider whether you plan on maintaining config files through a program (either the application being configured or a separate config maintenance application), or if editing config files with a text editor is sufficient. Some applications maintain their config files in XML format for this reason, as there are many packages that can easily read and write XML files, as well as do basic syntax checking outside of the application being configured. Properties files can also be easily written, but there are many other formats that could be used. This can get tricky if you are trying to use an application to maintain config files when you are using a general purpose language for those files.

No matter what format you settle on for your config files, the same concerns discussed below apply regarding reloading the config.

Config Objects

In the approach described here we store in-memory configuration information in special Config objects that are separate from the operational objects that they configure. Defining separate Config objects gives us these benefits:

• It allows us to represent multiple configurations simultaneously. In particular, it allows us to load and operate on a configuration that is separate from the currently active configuration.
• It provides a convenient location to collect the methods that manipulate or otherwise access the configuration parameters.

There should be a set of Config objects that correspond to the different operational objects that can be configured. Each different class of operational object to be configured should have a different custom class of Config object associated with it. An operational class with multiple instances should have a separate instance of its Config class associated with each operational instance.

The various Config objects should be related to each other in the same way as the operational objects are related to each other; for example, if operational object A can have multiple children of type B, then ConfigA should be able to have multiple children of type ConfigB. There should be a single Config object which serves as the root Config object from which all other Config objects can be reached.

If the application is written such that there is a single application-wide active configuration, then the application should have a singleton which is the active root Config object. In the discussion below, I assume that such a singleton exists; if your application has multiple contexts, each with a different set of config info, you should interpret the word "singleton" to refer to the single active root config for the context whose config is being updated.

All of the Config classes can inherit from a standard base Config class that provides implementations of common useful methods such as type-safe calls to get integer and date parameters.

Seven Steps

There are seven steps involved in loading or reloading configuration data: Trigger, Locate, Load, Validate, Activate, Report, and Use. Each of these steps can be considered independently of the others. Each step has its own design decisions and implementation choices. In the approach we are using, the Config objects mentioned above are the common data shared by all but the first two steps.

Trigger

If your application is going to reload its configuration information, it needs to know when to do that. There are a number of options:

• Your app can check for changes on a regular interval and reload if the source has changed. This is a typical approach used with logging configuration files such as for log4j, in which you can specify automatic reloading with a call to the static configureAndWatch method of DOMConfigurator or PropertyConfigurator.
• If your app has a command line interface (CLI), you can add a command that reloads the config info.
• If your app has a web interface, you can add a web page that controls config reloads. This can be a full web page with a form and feedback, or a simple URL that triggers a reload.
• On a Unix system, a standalone app such as a daemon can be written such that a reload is triggered on receipt of a signal. You can then use the kill command to send the process that signal. SIGHUP (signal 1) is often used by Unix daemon programs for this purpose, some examples being acpid, dnsmasq, postgresd, smartd, smbd, winbindd, and ypbind.
• For a Java app, you can enable JMX and use that to send commands to your application with a JMX console app such as jconsole or MC4J. JBoss uses this technique, allowing you to reload its log4j config using the JBoss jmx-console.
• For many apps, you can pretty easily add a web interface, such as by using Jetty for Java apps, for the purpose of allowing control and status feedback.

You may want to limit how often a reload can be triggered to prevent a DOS attack (or the same effect caused by a bug in whatever is producing the trigger).

Locate

Once the app has been triggered to reload the config info, it needs to locate that info. Some options:

• Assume the data is in the same location as before and reopen that location, such as is often done for a log4j config file.
• Provide the location of the data along with the trigger. This is easy to do if you have a CLI, web form, or web URL, not so easy if you are using a timer or a Unix signal.

Some applications (such as one that uses the standard Props class in Lift, including the way Lift handles its log4j configuration) have more sophisticated file lookup mechanisms that allow configuration information to be split among multiple files or segregated according to the runtime environment to be used. If you are using a package that looks for one or more out of a set of possible files, and you want to be able to add or remove a config file and then reload, you should check to make sure the package is able to reload files and that it will rescan its set of possible files and not just assume that the same config files should be used as when they were first loaded.

Load

Once the data has been located, it needs to be loaded into memory where it can be manipulated. Note that you should load the data into a new Config object or set of objects so that you can do the validation checks on it before activating it.

You should not have to write the code that actually loads the data, as there are a number of usable options available. As an example, you can store your config data in the standard Java Properties format, then load that data using Properties.load. After reading the data into a Properties object, you can create your custom Config objects from the data in the Properties object.

Validate

Once the config data is loaded into your Config objects, you are ready to validate the new configuration. You should make the following checks:

1. Ensure that the syntax of all configuration values is correct. Depending on how you loaded the data and converted it to your Config objects, some of these checks may already have been done. If there are any values which have not yet been checked for correct syntax, those values should be checked now.
2. Perform semantic checks on individual parameters. This includes things such as checking that numbers are within allowable ranges, or that each selection parameter has a value that is one of the allowable selections for that parameter.
3. Perform validity checks on multiple parameters. This includes situations in which you have two or more parameters that are related and which thus must have values consistent with each other.
4. Compare the new set of Config objects against the current set to ensure that all proposed changes are allowed. You may decide that some changes are too much work to bother to implement; you can disallow those changes in this step.

With a Config class that corresponds to each configurable operational class, we can put the validation code directly in those classes rather than in the operational classes.

After completing the above validation steps, and assuming there were no errors, you have done all error checking and know that you will be able to switch to the new config without errors, but you have not yet done so.

Errors in any of these steps should be collected so that they are available for Reporting.

Activate

Assuming that the loaded Config objects pass all of your validation tests, it is time to activate the new Config. While conceptually simple, this is the trickiest step.

The key issue here is ensuring that the application works properly in the presence of concurrent access to the config data. You want to make sure that the application cleanly switches from using the old configuration to using the new one, without the possibility that some operations will be performed with part of the old configuration and part of the new one.

There are two basic updates you need to make, which correspond to the two basic approaches to using the data:

1. Update the active root Config singleton.
2. Update all operational objects that contain configuration state.

Handling the first approach is pretty easy: inside a synchronized block, update the active root Config singleton. When another thread begins an operational sequence that relies on any config parameters, it reads the current root Config singleton (in a synchronized block) and keeps it in a local variable for the duration of the operational sequence. All queries for config parameters during that sequence are done against the local Config variable, ensuring that the entire sequence uses a single Config even if the Config singleton is updated in the middle of that operational sequence.

If you are using the second approach, updates are a bit trickier. It would be simple if the activation thread could just update the state in the operational objects, but another thread may currently be running and using those operational objects in an active operation. You can't just update the state in all of the operational objects from the activation thread because the operational thread might then pick up the new state in the middle of one operational sequence, and we assume that starting an operational sequence with one state and finishing it with another state will cause problems.

The key to handling changes when using this second approach is to build on how we solved changes to the first approach by capturing the value of the active root Config singleton at the start of the operational sequence. That starting point is the point at which we know (by definition) that it is safe to change over to a new config. When we start our operational sequence, we capture the currently active Config into a local variable, as described above as the solution for changes to the first approach. We then check to see if the config has changed since the last time we started the sequence. We do this by comparing our newly captured Config against the Config that we used the previous time we executed our sequence, which means we need a second variable that stores that previous Config. If the newly captured Config is not the same as the previous Config we used, then we reconfigure our operational objects according to the newly captured Config, then save that as well as the most recently used Config for the next execution.

When using the above solution, if the only time you update the operational state is when a thread starts an operational sequence, and that thread waits for a long time before beginning execution of the sequence, then the switch of the operational state to the new config may not happen for a long time. Despite having validated our new config, it is possible that, due to a bug, the new config will fail when we attempt to apply it to our operational objects, and it is generally better to have that happen immediately when the config is activated rather than much later, when it might not be obvious that the problem is due to the new config. In order to avoid this situation, you should add code to make your threads wake up and apply the new config immediately after it is activated, even if there is no other work for them to do.

If you have multiple independent operational sequences you should separately capture a copy of the active Config at the start of each sequence. However, you need to make sure that each sequence is in fact independent of the others as far as the config parameters that each uses, since when using the above approach you may end up with two threads executing different sequences at the same time with one using the old config and the other using the new config.

If the different threads are related, such that it is not acceptable for one thread to be using the new config while another is still using the old config, then you will have to use a different approach. In this case, you will probably need to write some code to ensure that no thread starts using a new config until all threads have stopped using the old config.

You can do this with two flags, properly synchronized:

1. config-in-use
2. ok-to-use-config

The activation thread turns off ok-to-use-config, then waits until config-in-use is zero. At that point it updates the root Config singleton and turns on ok-to-use-config.

The operational threads check ok-to-use-config before capturing the current config. If turned off, they wait until it is turned on. They then increment config-in-use, use the config, and decrement config-in-use when done. Synchronized and try/catch blocks should be used to avoid race conditions and ensure the config-in-use count doesn't get stuck on.

Report

Feedback is important. Ideally, the user should get the following feedback:

• When loading of an updated config is triggered, the user should get feedback on whether or not the new configuration was activated.
• If the new configuration was not activated, the user should get feedback on why the new configuration was rejected (i.e. he should see a list of config errors).
• Ideally, at a later point in time it should be possible for the user to determine what configuration is currently being used and how long it has been active. This is useful in situations where an on-disk config was changed at some point in the past but not loaded into the application.

Generally the reporting feedback channel is related to the trigger mechanism:

• If you use a CLI command to trigger the reload, that command can print out the feedback.
• If you use a web page to trigger the reload, the web response page can display the feedback.
• If you use JMX, the feedback can be returned through that protocol.
• If you use a web URL, the HTTP response can include the feedback.

If you application does logging, the feedback can be logged to the log file. This can be done in addition to any of the above feedback mechanisms.

Use

It is important to ensure that the config parameters used are consistent throughout an operational sequence, even when a config reload occurs while that sequence is executing, as discussed above in the Activate section above. Once you have handled that, you can move on to other usage aspects.

There are two basic approaches to using the active config parameters:

1. Use the current Config object directly each time a config value is needed. This is suitable for simple options that are tested each time a specific behavior or feature is desired.
2. Load data from the current Config object into operational objects. This is necessary when some of the config info refers to state that is managed by an operational object, such as the endpoint for a TCP connection.

The first approach provides for simpler updating of the config data, but sometimes the second approach is necessary for performance reasons or due to how state information is stored in other objects. The timing for when to update config state in operational objects is discussed in the Activate section above.

A minimal implementation of the Config object would provide just a single method to retrieve any parameter by name, such as is provided by the Properties.getProperty method. While this is easy to implement, it does not provide as much protection against programming errors as other approaches described below.

For type safety, you should implement (or use a package that provides) a set of methods with specific return types that match the types of your config parameters. You can then pass in the name of each parameter and not have to type-cast the result.

For maximum type safety your Config object should provide methods specific to each config parameter being retrieved. This ensures not only that you have the correct return type for the parameter, but that you have not accidentally mistyped a parameter name in a call to retrieve its value. (Of course, your unit tests should also catch this error, but you will catch it sooner and more surely with compile-time checks.)

Unit Testing

Keeping the configuration management code in separate Config objects improves the testability of your code. You can write unit tests for your Config objects to test that they properly locate, load (or reload), and validate config files, and you can create a set of mock Config objects that you can use to test how your application responds to different configurations.

A more complete test suite will include tests that verify proper functionality when a reload operation is performed by one thread while one or more other threads are in the middle of processing and using config data. However, a detailed discussion of this kind of multi-thread testing is beyond the scope of this post.

Implementation Options

You can write all of your own config code from scratch, or you can leverage an existing package. Whatever approach you take, you will want to ensure that your application handles all seven of the steps discussed above.

A few packages are listed below, with a discussion of the steps for which they provide support. For bullet items marked no support you will have to write your own code. None of the packages provides support for all of the steps. Even if a package did provide that support, you must still provide application-specific code for validation, activation, and use.

Caveat: Except for Properties, I have not used the packages listed below. My evaluation of their capabilities is based entirely on reading the documentation and examining the source code, so it is possible that I have made some mistakes in that evaluation.

• Properties (Java)
• JavaConfig (Java)
• Apache Commons Config (Java)
• Configgy (Scala)

Properties (Java)

The standard Java library includes the Properties class, which can be used for simple applications that require only a few parameters.

• Trigger: no support.
• Locate: no support.
• Load: You can load a properties file with a single call to Properties.load, where you pass in the name of the file to load.
• Validate: no support.
• Activate: no support.
• Report: no support.
• Use: The Properties.get method will return the value of a property as a String. You can use this directly as a generic call to retrieve config parameters by name, or you can layer your type-safe methods on top of this.

JavaConfig (Java)

JavaConfig (not to be confused with Spring JavaConfig, which is used for Dependency Injection configuration) reads config files using the standard Properties file format. The package provides a generic Config class, which you subclass to create your application-specific config class. It handles a defined set of data types.

JavaConfig specifically does not include any logging, so that it can be used to read the configuration for another logging package.

• Trigger: no support.
• Locate: no support.
• Load: You pass the name of a properties file to the Config constructor, which loads the properties file.
• Validate: After instantiating your config object, you call the validateConfiguration method on it, which returns a ConfigValidationResult object that contains the validation results. This validateConfiguration method calls all of your getter methods. For each of your methods that throws an exception, the message is collected and made available through the ConfigValidationResult object.
• Activate: no support.
• Report: The ConfigValidationResult class collects the error messages from all of your getter methods that throw exceptions, and makes them available
• Use: The base Config class provides type-safe methods such as getInt and getBoolean that accept a parameter name. In your config class that extends that class, you define a getter method for each of your config parameters. Each of your methods should call one of the underlying type-safe methods, passing it a config parameter name, and return that result. Your method should also perform any validation checks and throw an exception if there are any validation errors.

Apache Commons Config (Java)

Apache Commons Config provides a mechanism to allow config info to be loaded from a variety of sources, such as files or databases. You can mix config info from multiple different sources, such as reading some info from a database and some from system properties, and access it all through a single config object. It supports file includes and value substitution.

• Trigger: The package org.apache.commons.configuration.reloading provides a mechanism for defining a reload strategy when using file-based configuration, such as reloading on access to a config element if the file has changed, and some support for using JMX to trigger a reload.
• Locate: You can pass in a relative filename, and the package will look in various locations for a config file of that name to load.
• Load: You can create a configuration object for a specific data source, such as a file, or you can create a composite configuration object from multiple other configuration objects.
• Validate: no support.
• Activate: no support.
• Report: no support.
• Use: There are a set of type-safe methods to which you pass an item name and receive back its value.

Configgy (Scala)

Configgy includes logging as well as configuration, so it can use its own config files to configure logging. Its config files look like a cross between an XML file and a Properties file, with hierarchy represented by XML syntax, and individual parameters looking more like Properties. It handles a defined set of data types and has the ability to represent lists of values for a single parameter.

Configgy supports a lot of options when defining parameters, including hierarchy, inheritance, includes, variable substitution (including system properties), and conditional assignment.

Note that Configgy allows the application to set values in the config after it has been loaded into memory. As with any situation in which one datastructure might be shared among multiple threads, you should be very cautious with this capability. In particular, if you have a thread which has read the config and used that data to set state in its own application objects, setting the value in the config object alone may not have the desired effect. Your application can set up a subscriber for changes, which will be called when there are runtime changes to a config value, but you need to handle synchronization of these runtime changes in the same manner as when reloading the entire config. And your application code that is calling the set method must be prepared to handle a thrown exception if a subscriber decides the change is invalid.

The examples given in the Configgy documentation use a Scala object (as opposed to class) and does not discuss reloading, but there is a reload method available on the main object, which should work if you use the approach described above (using a pair of flags) for the case when all threads are related. Also, there are separate config objects being used under the covers, so it should be possible to use those directly, rather than the main object, if you want to be able to switch some threads over to your new config while some other threads continue to use the old config.

If you are writing a Scala application, Configgy is probably your best option.

• Trigger: There is some JMX support built in; reload is not one of the methods available from the JMX interface, but it should not be too difficult to add it.
• Locate: Configgy has calls to allow you to set the location of the config file to load. You can call this before calling reload() to control the source for the reload.
• Load: You call Configgy with a filename and it loads that file and any file referenced with an include statement.
• Validate: Configgy uses a subscription/callback model to let the application know when data has been changed. Your callback is called with an argument that tells you whether Configgy is doing a validation pass or an activation ("commit") pass. On the validation pass, your callback can throw an exception to indicate that the new value fails validation.
• Activate: The validate/commit subscription model provides hooks to allow you to write your own validation and activation, but you still need to consider synchronization when using multiple threads.
• Report: no support.
• Use: There is a set of type-safe methods to which you pass a parameter name, which can be hierarchical.

Unit Testing with Dates and Times

When running unit tests or regression tests, you should know exactly what all of your inputs are and be able to control them. If you have methods that use the current date or time, you need to do a bit of extra work to do that.

Contents

• Refactor
• Test Hook
• Controlled Values
• Changing Values
• Capturing Values
• Demonstration

Refactor

If you have a method that calls System.currentTimeMillis() or new java.util.Date(), that method will be receiving a different value when you run it on a different day or time. This contradicts the need to have exact control over all inputs for unit and regression testing. You need to refactor your method so that you can control the date or time value used by your method and thus be able to have repeatable and controllable tests.

To start, create a utility class with a method corresponding to each of the time-variant methods System.currentTimeMillis(), new Date(), or any other method whose value is a function of the current date and time.

import java.util.Date;
public class TimeVariant {
    public static long currentTimeMillis() {
        return System.currentTimeMillis();
    }

    public static Date newDate() {
        return new Date();
    }
}

Modify your code to replace all calls to new Date() by calls to TimeVariant.newDate(), replace all calls to System.currentTimeMillis() by calls to TimeVariant.currentTimeMillis(), etc.

If your application uses a database and takes advantage of any automatic timestamping by the database server, you will have to figure out a way to deal with those values. I'm not going to try to cover unit testing with a database; that is a big enough topic to warrant a separate post.

Upon completing this step, you have not changed any of the functionality of your code, but you have centralized all of the time-variant calls into a single location. Now we can modify the TimeVariant class to make it do what we want.

Test Hook

The general goal is to have methods in TimeVariant that behave as defined above during normal operation, but for which we can control the return value during testing. In order to keep the testing code out of the main code, we want to make minimal changes to TimeVariant and put as much of the support code as possible into another class that we use only in our testing environment.

We start by defining an interface that duplicates the functionality of the TimeVariant class:

import java.util.Date;
public interface ITimeVariant {
    public long currentTimeMillis();
    public Date newDate();
}

We will be writing a test class TestTimeVariant that implements this interface. If you are already using an injection framework such as Spring, you can modify our original TimeVariant class so that it implements the ITimeVariant interface with non-static methods, make your code use the instance of ITimeVariant supplied by your injection framework, and configure the framework to use an instance of TimeVariant in normal use and TestTimeVariant during testing.

If you are not using an injection framework, one way to allow using the methods in TestTimeVariant during testing is to modify TimeVariant to allow turning it into a proxy for an instance of ITimeVariant. Changes from the previous version of this class above are shown in bold.

import java.util.Date;
public class TimeVariant {
    private static ITimeVariant tvDelegate = null;

    /** This method is for unit testing setup. */
    protected static void setDelegate(ITimeVariant tv) {
        tvDelegate = tv;
    }

    public static long currentTimeMillis() {
        if (tvDelegate!=null)
            return tvDelegate.currentTimeMillis();
        else
            return System.currentTimeMillis();
    }

    public static Date newDate() {
        if (tvDelegate!=null)
            return tvDelegate.newDate();
        else
            return new Date();
    }
}

From the setup method for your unit test you set the hook:

    ITimeVariant tv = new TestTimeVariant();
    TimeVariant.setDelegate(tv);

Once the hook is in place to invoke our test version of the time variant calls, we are done with the changes to the production code and we can move on to the implementation of the test class.

This post uses Java for all of the code examples, but the same approach as described here can be used in other languages (Ruby, C#), where their language features can make setting up the hook and delegate more elegant than in Java.

Controlled Values

Recall that the primary purpose of the test class is to ensure that we return a known and controlled value for each call to one of our methods. In the simplest case, we can just return a constant value, in which case our TestTimeVariant class might look like this:

import java.util.Date;
public class TestTimeVariant implements ITimeVariant {
    private long simulatedTime = 1234567890L*1000;        //a recent timestamp

    public long currentTimeMillis() {
        return simulatedTime;
    }

    public Date newDate() {
        return new Date(simulatedTime);
    }
}

The above code satisfies our desire to have a controlled value, and it may be perfectly satisfactory for testing simple methods that call newDate() and use only the date portion or that call currentTimeMillis() only once, but it can be a bit unrealistic when our methods are called more than once.

Changing Values

We can modify the methods in our TestTimeVariant class to return different values on each call.

To start, we add a method that allows us to set the simulated current time. This allows us to set the date and time to specific values in order to test corner cases on our code, such as February 29 or December 31 on a leap year, or whatever other conditions are considered special by the method under test.

import java.util.Date;
public class TestTimeVariant implements ITimeVariant {
    private long simulatedTime = 1234567890L*1000;        //a recent timestamp

    public void setNow(long t) {
        simulatedTime = t;
    }

    public long currentTimeMillis() {
        return simulatedTime;
    }

    public Date newDate() {
        return new Date(simulatedTime);
    }
}

With the setNow test method, we can create a unit test that alternately calls production code and changes the time. This is suitable when testing methods that contain a single call to one of our time methods, but if a method makes more than one call before returning, it will get the same return value every time.

To return different values when there are multiple calls where we are not able to call setNow between then, we modify our test code to add a fixed delta to the returned value on each call.

import java.util.Date;
public class TestTimeVariant implements ITimeVariant {
    private long simulatedTime = 1234567890L*1000;        //a recent timestamp
    private long deltaTime = 10;

    public void setNow(long t) {
        simulatedTime = t;
    }

    private void bumpTime() {
        simulatedTime += deltaTime;
    }

    public long currentTimeMillis() {
        long t = simulatedTime;
        bumpTime();
        return t;
    }

    public Date newDate() {
        return new Date(currentTimeMillis());
    }
}

If we want more precise control, we can add the ability to preset a list of values to be returned, or equivalently a list of delta values to be added on each call:

import java.util.Date;
public class TestTimeVariant implements ITimeVariant {
    private long simulatedTime = 1234567890L*1000;        //a recent timestamp
    private int simCount = 0;
    private long[] deltaTimes = { 10 };         //default delta

    public void setNow(long t) {
        simulatedTime = t;
    }

    public void setDeltaTimes(long[] deltas) {
        if (deltas==null || deltas.length==0)
            throw new IllegalArgumentException("Must have at least one delta");
        deltaTimes = deltas;
            //Could make a copy if you are concerned about caller changing it
        simCount = 0;
    }

    private void bumpTime() {
        simulatedTime += deltaTimes[simCount];
        if (simCount < deltaTimes.length - 1)
            simCount++;
    }

    public long currentTimeMillis() {
        long t = simulatedTime;
        bumpTime();
        return t;
    }

    public Date newDate() {
        return new Date(currentTimeMillis());
    }
}

We have defined our bumpTime method so that if we call one of our time methods more times than we have deltas, we just reuse the final delta for any additional calls. Depending on your desires, you could use other approaches, such as cycling back to the beginning of the list of deltas or using an algorithmic approach. For example, you could choose to use a pseudo-random number within some range as your delta, and as long as you seed your random number generator with the same seed each time you will get the same sequence of time values.

Capturing Values

Likely a more realistic series of time values is a sequence of time values from an actual run of your method. We can instrument our test class so that we can capture all of the returned time values during a test run, then play them back for our controlled testing. This adds a significant amount of additional code to our test class.

You may not need this level of functionality for your tests. In fact, you may need nothing more than returning a single fixed value, as shown in the Controlled Values section. There is no need to implement more functionality in the test code than you need for your situation, so you should think about why you need changing values in the first place. Note that none of this code is useful for any kind of timing test, since it does precisely the opposite: no matter how much real time elapses between calls, the same values are returned from the methods in TestTimeVariant. This is exactly the right behavior for repeatability, and exactly the wrong behavior for ensuring that the timing (and thus the performance) of your application has not changed. Running timing and performance tests on your application is a separate concern from running the type of functional unit and regression tests we are concerned with here.

import java.io.FileReader;
import java.io.LineNumberReader;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.Date;

public class TestTimeVariant implements ITimeVariant {
    private boolean capturing = false;
    private ArrayList<Long> captureTimes = null;
    private long simulatedTime = 1234567890L*1000;        //a recent timestamp
    private int simCount = 0;
    private long[] deltaTimes = { 10 };         //default delta

    public void setNow(long t) {
        simulatedTime = t;
    }

    public void setDeltaTimes(long[] deltas) {
        if (deltas==null || deltas.length==0)
            throw new IllegalArgumentException("Must have at least one delta");
        deltaTimes = deltas;
            //Could make a copy if you are concerned about caller changing it
        simCount = 0;
    }

    //Starts capture mode.  Call saveCapture when done.
    public void startCapture() {
        capturing = true;
        captureTimes = new ArrayList<Long>();
    }
    
    //Save the captured timestamps to a file for later loading by loadCapture.
    //The first line is the fill timestamp, the rest are deltas.
    public void saveCapture(String fileName) throws Exception {
            //You can use more precise exception list if you want
        PrintWriter pw = new PrintWriter(fileName);
        long lastTime = 0;
        for (Long t : captureTimes) {
            long delta = t - lastTime;
            pw.println(delta);
            lastTime = t;
        }
        pw.close();
    }

    //Load a file created by saveCapture.
    //Sets simulatedTime and deltaTimes array.
    public void loadCapture(String fileName) throws Exception {
            //You can use more precise exception list if you want
        LineNumberReader lr = new LineNumberReader(new FileReader(fileName));
        String line = null;
        boolean haveInitial = false;
        ArrayList<Long> dTimes = new ArrayList<Long>();
        while ((line=lr.readLine())!=null) {
            long d = Long.valueOf(line);
            if (haveInitial)
                dTimes.add(d);
            else {
                simulatedTime = d;
                haveInitial = true;
            }
        }
        //Transfer into long[] to cal setDeltaTimes
        long[] da = new long[dTimes.size()];
        int n = 0;
        for (long t : dTimes)
            da[n++] = t;
        setDeltaTimes(da);
    }

    private void bumpTime() {
        simulatedTime += deltaTimes[simCount];
        if (simCount < deltaTimes.length - 1)
            simCount++;
    }

    public long currentTimeMillis() {
        if (capturing) {
            long now = System.currentTimeMillis();
            captureTimes.add(now);       //boxed
            return now;
        } else {
            long t = simulatedTime;
            bumpTime();
            return t;
        }
    }

    public Date newDate() {
        return new Date(currentTimeMillis());
    }
}

Demonstration

Here is a little Main program that shows how the above classes work.

public class Main {
    public static void main(String[] args) throws Exception {
        TestTimeVariant tv = new TestTimeVariant();
        if (args.length==0)
            appMethod();        //run without any test code
        else if (args[0].equals("-test")) {
            TimeVariant.setDelegate(tv);
            long[] zeroDelta = { 0 };
            tv.setDeltaTimes(zeroDelta);  //simulate fixed-time case
            appMethod();
        }
        else if (args[0].equals("-delta")) {
            TimeVariant.setDelegate(tv);
            //without setting any deltas, default is 10ms
            appMethod();
        }
        else if (args[0].equals("-save")) {
            TimeVariant.setDelegate(tv);
            tv.startCapture();
            appMethod();
            tv.saveCapture(args[1]);
        }
        else if (args[0].equals("-load")) {
            TimeVariant.setDelegate(tv);
            tv.loadCapture(args[1]);
            appMethod();
        }
        else {
            System.out.println("unknown argument "+args[0]);
            //unrecognized
        }
    }

    private static void sleep(int ms) {
        try { Thread.sleep(ms); } catch (Exception ex) {}
    }

    //This is the application method under test
    public static void appMethod() {
        long startTime = TimeVariant.currentTimeMillis();
        sleep(100);
        long midTime = TimeVariant.currentTimeMillis();
        int n = 2;
        for (int i=0; i<1000000; i++) {
            n = n * n + i;              //lots of overflows
        }
        long endTime = TimeVariant.currentTimeMillis();
        System.out.println("App: start at "+startTime);
        long delta1Time = (midTime - startTime);
        System.out.println("App: delta1 is "+delta1Time);
        long delta2Time = (endTime - midTime);
        System.out.println("App: delta2 is "+delta2Time);
    }
}

To run this demo, copy out the definitions for ITimeVariant and TimeVariant from the Test Hook section above, the definition for TestTimeVariant from the Changing Values section above and Main from this section each into an appropriately named file in a test directory. Compile all files, then run as follows to see the different cases:

1. Run java Main to run the application as it normally runs, without any test code. The application prints out some timing values.
2. Run java Main -test to wire in the test harness with no delta times, which behaves the same as the first example in the Controlled Values section above.
3. Run java Main -delta to wire in the test harness with a fixed delta value, as you would have with the first example in the Changing Values section above.
4. Run java Main -save filename to wire in the test harness to use real times and save the results to the specified file. The file is a text file, so you can view it to confirm the values it contains.
5. Run java Main -load filename to wire in the test harness and load the values from the specified file.

Java Resources Support

One of the programming philosophies that I try to follow in order to encourage myself always to do my best work is to assume that every program I write will eventually become a "production" program, distributed to customers all over the world. One detail that falls into that "all over the world" part is internationalization. In other words, I design internationalization into my programs from the start, even when the program is primarily for internal or personal use.

Java makes the string-replacement part of this pretty easy to do with its ResourceBundle class. If you use ResourceBundle consistently, you can easily ship localizations for multiple languages bundled with your application, and other people can take your program and resource files and localize them for other languages without having to recompile your program.

If you have thousands of strings being translated into dozens of languages, you might want to get yourself a sophisticated translation management system or ship around XLIFF files (or perhaps you would like to help write an open source localization repository). But for smaller projects such as I have worked on, I have found that some relatively simple support methods and a bit of convention have served me pretty well.

Contents

• Building The Resources
• The Basic Interface
• The Basic Implementation
• Field Names In Format Strings
• Runtime Overriding Of Resources
• Recursive Resources

Building The Resources

To simplify loading resources, I find it easiest to put them all into one properties file per locale. But to improve maintainability, it is best to keep the resources in multiple files close to the source files in which they are used. I bridge these two conflicting goals by keeping my resources in multiple properties files, then concatenating them all together into a single properties file when I build my application.

To avoid confusion and emphasize the fact that my properties source files are not used in their source file form, I use the extension .props for those files. My concatenation program collects all of the .props files for one locale into one .properties file. I run it once for each locale for which I have files. The concatenation program also adds separators and source filenames in comments between the contents of each source file, and converts non-ascii characters to backslash-u notation. While I am at it, I also stuff some other information into the generated properties files, such as version and date information, so that I can easily access those details at runtime.

In conformance with the standard per-locale naming convention for resource bundle properties files, for each .props base file there can also be locale-specific translations. For example, for the base file Foo.props there could be Foo_hu.props with Hungarian translations, Foo_fr.props with French translations, and Foo_fr_CA.props with French-Canadian translations.

The properties file with the collected resources goes into a file in the same directory as the class files and gets packaged up in the jar file along with the class files. For example, I might name the collected resources file net/jimmc/app/Resources.properties, so it goes in the directory with all of the files in the net.jimmc.app package. When I load it, I specify its location as net.jimmc.app.Resources and the ResourceBundle code automatically finds it from my jar file, as shown in the initResources method below.

The Basic Interface

In order to be able to use the simplest possible code when calling to retrieve resources in my Java apps, I use support code, shown below, to do the following:

• Encapsulate my ResourceBundle in the support code so that the caller does not need to worry about where the resources come from.
• Make my resource retrieval functions so that, when a key is not found in the resources, it returns the key as the value of the method rather than throwing an exception. The key then appears in my GUI, where I can see it and correct it later when I want, rather than causing an exception that stops the program and forces me to correct the string immediately before I proceed.
• Retrieve a resource string and format it with one or more arguments in one call.

In my Java apps, I define a ResourceSource interface with definitions for methods to implement the above functionality:

public interface ResourceSource {
    /** Get a string from resources.
     * @param name The resource name.
     * @return The value of the resource,
     *         or the name if no value is found.
     */
    public String getResourceString(String name);

    /** Get a string from resources, or null if not found.
     * @param name The resource name.
     * @return The value of the resource,
     *         or null if no value is found.
     */
    public String getResourceStringOrNull(String name);

    /** Get a string from a resource and pass it to MessageFormat.format
     * with the specified arguments, returning the result.
     * @param name The resource name.
     * @param args The args to MessageFormat.format.
     * @return The formatted resource string.
     */
    public String getResourceFormatted(String name, Object[] args);

    /** Get a string from a resource and pass it to MessageFormat.format
     * with the specified one argument put into a new Object[1],
     * returning the result.
     * @param name The resource name.
     * @param arg The argument to MessageFormat.format.
     * @return The formatted resource string.
     */
    public String getResourceFormatted(String name, Object arg);
}

The Basic Implementation

My Application object, a singleton, implements my ResourceSource interface and is the source for resource values for the application. (This is conceptually the same as the IResources interface I mentioned in a previous post.) It contains an internal reference to my resources, so that the caller does not need to worry about that:

import java.text.MessageFormat;
import java.util.ResourceBundle;
    private ResourceBundle resources;

During initialization of my Application object, I call a simple initResources method to load my resources from that one file containing the concatenated contents of all of the props files:

    private void initResources() {
        resources = ResourceBundle.getBundle("net.jimmc.app.Resources");
    }

My Application object has a private method that accesses my resources, which all of my other Application resource methods call (to simplify adding other functionality later):

    private String getResourceValue(String name)
            throws MissingResourceException {
        return resources.getString(name);
    }

The methods in the Application object that implement the ResourceSource interface all call getResourceValue to get the resource value:

    public String getResourceString(String name) {
        try {
            return getResourceValue(name);
        } catch (MissingResourceException ex) {
            return name;        //use the resource name
        }
    }

    public String getResourceStringOrNull(String name) {
        try {
            return getResourceValue(name);
        } catch (MissingResourceException ex) {
            return null;
        }
    }

    public String getResourceFormatted(String name, Object[] args){
        String fmt = getResourceString(name);
        return MessageFormat.format(fmt,args);
    }

    public String getResourceFormatted(String name, Object arg) {
        String fmt = getResourceString(name);
        Object[] args = { arg };
        return MessageFormat.format(fmt,args);
    }

Field Names In Format Strings

When using the formatting methods in java.text.MessageFormat you specify the locations of replacement values using numbers in braces, so your formatting string might contain phrases such as {1} or {2,date,medium}. You have to ensure that these index numbers correspond properly to the arguments in the source code that uses that format string. Since these format strings are in a different file (the resource properties file) than the code that uses them, extra care is required to avoid mixing up the numbers, such as when adding or modifying the argument list. To minimize this problem, I add a version of getResourceFormatted that allows using field names rather than numbers in the format strings, so they contain phrases such as {count} and {modtime,date,medium}. When calling these methods, the fieldNames array contains names that correspond to the args of the same index.

    /** Get a string from the resource file, map fields names to numbers,
     * and format it with the given arguments.
     */
    public String getResourceFormatted(String name, Object[] args,
                String[] fieldNames){
        String fmt = getResourceFormat(name,fieldNames);
        return MessageFormat.format(fmt,args);
    }

    /** Get a resource string and map field names to numbers. */
    public String getResourceFormat(String name, String[] fieldNames) {
        String fmt = getResourceString(name);
        return mapFieldNamesToNumbers(fmt, fieldNames);
    }

    /** Replace named MessageFormat field references by field numbers.
     * The named field reference looks just like a regular format
     * segment, but with a string in place of the number.
     */
    public String mapFieldNamesToNumbers(String s, String[] fieldNames) {
        for (int i=0; i<fieldNames.length; i++) {
            String fieldName = fieldNames[i];
            if (fieldName==null || fieldName.equals(""))
                continue;       //skip blanks in the array
            String p = "\\{"+fieldName+"\\}";
            String q = "{"+i+"}";
            s = s.replaceAll(p,q);
            p = "\\{"+fieldName+"\\,";
            q = "{"+i+",";
            s = s.replaceAll(p,q);
        }
        return s;
    }

Runtime Overriding Of Resources

In order to allow loading additional resources at run time, for debugging, I add an additional variable to store my override resources:

    private Properties resourceProperties;

I then add a method, which gets called from a debugging command to which I have specified a resources properties file name, to load values from the specified properties file into my override resources:

    public void loadResourceProperties(String filename) {
        if (resourceProperties==null)
            resourceProperties = new Properties();
        try {
            FileInputStream ifile = new FileInputStream(filename);
            resourceProperties.load(ifile);
            String msg = getResourceFormatted("info.LoadedResourceProperties",
                    filename);
            startupMessage(msg);
        } catch (Exception ex) {
            String msg = getResourceFormatted(
                    "error.LoadingResourceProperties",filename);
            throw new RuntimeException(msg,ex);
        }
    }

I modify my private getResourceValue method to look in my override properties before using the regular properties:

    private String getResourceValue(String name)
            throws MissingResourceException {
        if (resourceProperties!=null) {
            String v = resourceProperties.getProperty(name);
            if (v!=null)
                return v;       //found an override property value
        }
        return resources.getString(name);
    }

Recursive Resources

For some applications I want to be able to define resource strings that reference other resource strings or context values provided by the application. The standard replacement text in a message format starts with an open brace followed by a number. In my resource files, after implementing my Field Name extension above, I can also follow an open brace by a field name. With the extension in this section, I can follow an open brace by the "@" character and a context name or the name of another resource. The values for the context names are passed to the formatting methods in a Map.

For example, if I have these resources:

Sample.abc.foo=A formatted string with {@Sample.abc.bar} for {@User}
Sample.abc.bar=a nested property

and I create a context Map with User=Jim, then using the methods in this section I can call getResourceFormattedRecurse passing name as "Sample.abc.foo" along with my context and get back "A formatted string with a nested property for Jim".

To implement this capability, I add the following to my ResourceSource interface:

    /** Get a string from resources, recursing to do replacement of
     * special patterns with other resources.
     */
    public String getResourceStringRecurse(String name);

    /** Get a formatted string from the resource file, including
     * recursive replacement of special strings.
     */
    public String getResourceFormattedRecurse(String name, Object[] args,
            String[] fieldNames, Map ctx, int maxRecurse);

I add the corresponding code to my Application object:

    public final static int DEFAULT_MAX_RESOURCE_RECURSE = 20;

    /** Get the value of a resource, replacing the special tags.
     * @see #replaceResourceRecurse
     */
    public String getResourceStringRecurse(String name) {
        String value = getResourceStringOrNull(name);
        if (value==null)
            return name;
        return replaceResourceRecurse(value);
    }

    /** Get a formatted string from the resource file, including
     * recursive replacement of special strings.
     */
    public String getResourceFormattedRecurse(String name, Object[] args,
            String[] fieldNames, Map ctx, int maxRecurse) {
        String fmt = getResourceString(name);
        if (maxRecurse>0)
            fmt = replaceResourceRecurse(fmt, ctx, maxRecurse);
        if (fieldNames!=null)
            fmt = mapFieldNamesToNumbers(fmt, fieldNames);
        return MessageFormat.format(fmt,args);
    }

    /** Given a string, look for occurrences of our special replacement
     * syntax and replace with the referenced value, with a
     * default recursion limit.
     * @param s The string through which we look for our special
     *          replacement tags.  The replacement tags are of the
     *          form {@XXX} where XXX is the name of the field with
     *          which to replace that tag.
     * @throws RecursionLimitException if the maximum recursion depth
     *         is exceeded.
     * @see #replaceResourceRecurse(String,Items,int)
     */
    public String replaceResourceRecurse(String s) {
        return replaceResourceRecurse(s, null, DEFAULT_MAX_RESOURCE_RECURSE);
    }

    /** Given a string, look for occurrences of our special replacement
     * syntax and replace with the referenced value.
     * @param s The string through which we look for our special
     *          replacement tags.  The replacement tags are of the
     *          form {@XXX} where XXX is the name of the field with
     *          which to replace that tag.
     * @param ctx The context dictionary for replacement.  We first look
     *          for XXX in this dictionary; if not found, we look for
     *          a resource named XXX.  May be null.
     * @param maxRecurse The maximum recursion depth.
     * @throws RecursionLimitException if the maximum recursion depth
     *         is exceeded.
     */
    public String replaceResourceRecurse(String s, Map ctx, int maxRecurse){
        if (s==null)
            return s;
        int x = s.indexOf("{@");        //look for our replacements
        if (x<0)
            return s;   //no replacements
        StringBuffer sb = new StringBuffer();
        int a = 0;
        while (x>=0 && x<s.length()) {
            sb.append(s.substring(a,x));
            int y = s.indexOf("}",x);
            if (y<0) {
                String eMsg = "Missing close brace "+
                    "in resource after "+
                    s.substring(x,s.length());
                //TBD i18n
                throw new RuntimeException(eMsg);
            }
            String name = s.substring(x+2,y);
            String v = null;
            if (ctx!=null)      //try the context if we have one
                v = (String)ctx.get(name);
            if (v==null) {
                //If no context property, try resource
                v = getResourceStringOrNull(name);
            }
            if (v!=null) {
                if (maxRecurse<=0) {
                    throw new RecursionLimitException(name);
                }
                //Found the specified resource, recurse on it and
                //replace specials in it before we plug it into our string.
                try {
                    v = replaceResourceRecurse(v,ctx,maxRecurse-1);
                } catch (RecursionLimitException ex) {
                    //If we exceeded the recursion limit, we will be popping
                    //back through here many times; we add on the name of
                    //each item in the recursion chain to add in debugging.
                    ex.setMessage(name+"->"+ex.getMessage());
                    throw ex;
                }
                sb.append(v);
            }
            a = y+1;
            x = s.indexOf("{@",a);
        }
        if (a<s.length())
            sb.append(s.substring(a));
        return sb.toString();
    }

RecursionLimitException is a simple exception class that also allows setting the error message:

public class RecursionLimitException extends RuntimeException {
    private String ourMessage;

    /** Create a RecursionLimitException. */
    public RecursionLimitException(String message) {
        super(message);
    }

    public void setMessage(String msg) {
        this.ourMessage = msg;
    }

    public String getMessage() {
        if (ourMessage!=null)
            return ourMessage;
        else
            return super.getMessage();
    }
}

Command Line GUI Java Apps

Over the years I have written a number of GUI-based Java apps, first using AWT and later using Swing. Often at some later point I wanted to invoke the app from the command line to make it do something in batch mode with no user interaction. Below is the approach I now use when writing a GUI app in order to avoid having to modify the app later to make it work in batch mode. (The descriptions are a synthesis of some techniques I have used, and names are representative.) I have used this approach with both Java and Scala.

Implementation

The first rule is relatively common advice: separate the model from the view and controller (the MVC paradigm). Usually this is pretty easy, and once you get into the habit of doing this it goes a long way towards making it possible to use the application in a non-interactive mode. But there were always a few places where I wanted to output a message, or ask for confirmation, or do some other kind of I/O that would tie my code to the view.

My solution to this problem is to create an IBasicUi interface to handle I/O, an IResources interface to handle resources, an Application class that implements those interfaces for normal graphical use, and a BatchApplication class that implements those interfaces for non-interactive batch use. At runtime, based on the startup options, I instantiate an instance of either Application or BatchApplication as my Application object.

The Application object is responsible for three things:

1. Console output such as errors, warnings, questions and info.
2. Console input such as confirmations or file names.
3. Initializing Resources (resource bundles).

The GUI Application object uses popup boxes for console output and dialog boxes for input. The batch Application object just prints to System.out and reads input from System.in.

The Application object contains most of the methods that behave differently between GUI and batch use, although occasionally I will use a non-object-oriented approach by putting some of that in other classes and implementing an isBatch() method in the Application object to allow the other classes to change their behavior.

I also use the Application object to store other application-wide state, such as command line options that affect the behavior of the application.

Usage

I would typically pass a pointer to the Application object in to the constructors of my other objects. Alternatively, I sometimes would define an Application class and declare a static variable which I would set to the Application object on initialization, after which I could retrieve it with a static getter method. Finally, if for some reason neither of those two approaches was acceptable, I might put the Application object into a thread local variable and retrieve it from there when needed.

I suppose using today's lingo the act of passing in an Application object that could be either an Application or a BatchApplication would be referred to as dependency injection.

An additional benefit I got from having every object use the Application object for input, output and resources was that it made unit testing much simpler. I could create a TestApplication class in which the input and output methods were tied to my testing data files, and the resource methods used a small resource file with resources specific to the test being run.

Another Detail

By default when you start a Java application the JVM initializes the graphical environment. If you are running your batch command from your windowing system, this is not a problem; but if you are running it from a true batch script, where there is no windowing system available, this is not good. To prevent Java from initializing the graphical environment, you have to tell it that you are running in a headless environment. You do this by including the JVM option -Djava.awt.headless=true when starting your Java application. The app can tell if this command line option has been set by calling GraphicsEnvironment.isHeadless().