Build System Design
- Version:
- October 24, 2006
- Author:
- Jesse Glick, Jan Rojček
- Abstract:
-
For NetBeans 4.0 a new build system was introduced. An Ant-based
design was selected, with principal design goals including architectural
simplicity, ease of expansion, robust headless builds, good control by project
type of the UI, and an initial feature set adequate for everyday development.
This document outlines the fundamental design principles of that system.
Contents
First, an overview of how the build system works as far as the user is
concerned, without regard for its implementation in NetBeans modules. This
section describes the currently implemented UI and build behavior.
User development files (e.g Java sources) are grouped into logical units called
projects. Generally, a project is a disk folder containing some sources
of various kinds; some metadata describing its structure and contents for use by
the IDE; an Ant build script; some files (in Java
*.properties format) giving parameters needed by Ant as well as
the IDE; and build products created by Ant.
Each project has a type which specifies its behavior. For example, a
J2SE project is expected to contain some Java sources, perhaps some JUnit-based
test sources, perhaps a JAR manifest, etc.; its behaviors include compiling the
sources, creating a JAR file, running the program using a
main(String[]) method, debugging the program, running tests,
generating Javadoc, etc.
(Currently the project type is expected to anticipate and handle all aspects of
the project’s behavior, but see below for more.)
You can create new projects using wizards in the IDE’s UI and they should
be more or less ready to go when they are created—depending on the project
type, you may need to first add some source files. You can open or
close existing projects to display them in the IDE’s
Projects tab. (Whether a project is open or not does not affect much
besides the display of this tab and some global actions in the UI that need a
list of likely files to work on—you can freely edit files from projects
that are not open in the UI, build them, etc.)
Every file on disk may belong to a project (or it may not). But a file
may only belong to one project at a time. Normally all files beneath a project
directory are considered to belong to it. (A project directory may be contained
within another project directory—in this case files by default belong to
the nearest containing project directory.) The project which owns a file is able
to supply some behaviors for it—for example, a Java source file contained
in the Java sources folder of a J2SE project will have editor code completion
choices appropriate to that project.
The IDE’s UI supplies a set of actions (e.g. menu items) which perform
operations on the project. For example, if you select any file contained in a
project and choose Build from the UI, the project will be built. (Some
actions are context-sensitive, so that for example with a Java source
file selected you can choose Compile File to compile just that one source
file.) Each operation is typically performed by just running Ant.
So what does a project look like on disk? To start with, for the purposes of
this document, we will assume that your projects are kept under version control,
e.g. CVS. You can also have a project that is not under version control if you
like living dangerously. Unless otherwise stated, everything in the project is
considered versionable. See below for more details.
At the top level of your project directory there may be various source folders,
according to the project type. There may also be externally located source folders.
There is also a file build.xml, an Ant build script. This script is
the entry point for building your project—whenever you invoke a
build action from the IDE, a particular target in this script is run. You can
edit this script if you want, modifying or even replacing anything in it. But
the build.xml as created by the new project wizard is almost empty—it
just delegates all actions to an implementation script
which it imports (see below).
A subfolder called nbproject holds project metadata and the
implementation of the Ant build. A file project.xml holds general
project configuration. This includes the type of project, (perhaps) its name, a
list of references to subprojects (see below), and some basic structural
parameters of the project—e.g. which version of Ant it requires as a
minimum, whether the project build specifies a particular Java platform to use
instead of the default, etc. The XML file is extensible but may be
syntax-checked using XML Schema. It is normally modified via the
IDE’s GUI.
nbproject also contains an Ant build script called
build-impl.xml. The meat of the build semantics for the project is
defined here. You cannot directly edit this script; it is automatically
regenerated whenever project.xml changes. (See below for more on
how to customize the Ant build by hand despite this.) Although it is generated
from project.xml, it should normally be kept under version control
to ensure that users of command-line Ant, or other IDEs, can build the project
by just checking it out and running it. (More below.)
Also in nbproject is project.properties. This file (in
Java properties format) contains most build parameters, such as locations of
libraries, javac option settings, etc. It is normally updated by
the GUI when you modify project configuration, but you may also edit it by hand,
e.g. using the IDE’s text editor.
nbproject/genfiles.properties is used to detect upgrade situations. It should normally be kept in a VCS,
but should not be edited by hand, and if deleted will not cause much harm.
nbproject has a subdirectory called private that
contains any per-user customizations for this project. (Technically, this is per
checkout of the project rather than per user as such.) This directory is not
kept under version control. private.xml may contain transient
information, such as files the developer has opened from the project recently.
private.properties can set build parameters—any property
defined here overrides any definition in project.properties.
Finally, nbproject/private can contain a cache subdirectory
for internal use by the IDE. This directory, if it exists, is not versioned.
Currently this facility is not used.
Besides project.properties and private.properties, a
build script also loads settings from a file named build.properties
located in the IDE user directory. Typically this file would give the locations
of Java platforms, libraries, or other externally-installed resources. You can
keep these links in one place and refer to them by name in particular projects.
Typically entries in this file are regenerated from other IDE configuration data in the user
directory.
As is normal in properties files used by Ant, properties can refer to other
properties that have already been defined—including elsewhere in the same
file—using the ${propname} syntax. A build script will
typically load private.properties first, then the global
build.properties, and finally project.properties.
Earlier definitions always take precedence.
If you do not know how to write Ant scripts, or just do not wish to be bothered,
you can manipulate all commonly required aspects of the project and run common
build operations from the IDE’s GUI—you are not required to deal
with Ant directly. You can open a project customizer dialog and change the
classpath, JAR compression options, common Javadoc flags, etc. Changes are saved
to disk and so take effect the next time you run an Ant target.
If you do know how to use Ant, you can customize your project to your
heart’s content, all in build.xml. You can add some new
targets. You can perform additional steps before or after any predefined target.
You can even make a target not delegate to build-impl.xml,
and instead do something quite different: for example, if you have very specific
requirements for how to generate Javadoc, you can edit build.xml to
run the <javadoc> task exactly the way you like it, rather
than in the way the IDE set it up. You can use any tasks bundled with Ant, or
even tasks you create yourself or get from some external source.
If you make such customizations, the IDE will not “know about” them
in any real sense—that section of the project customizer dialog will not
be of any use to you; and design-time features like code completion may not be
completely accurate. Generally, the IDE only pays attention to
project.xml and the various properties files when it is deciding
how to let you work with a project in the GUI.
Update for NetBeans 6.0: see Project Configurations
When you select an action in the IDE’s GUI (such as Build) this
just runs a target of a predefined name (e.g. build) in your
build.xml. From there, the regular Ant integration in NetBeans
takes over. The target is run using the globally configured Ant
installation—by default, the copy of Ant 1.6.2 bundled with the IDE, but
you can choose another installation (at least version 1.6.0 is required for most project types). The IDE loads and
runs Ant in the IDE’s JVM.
Ant prints various messages about its progress, which are sent to the Output
Window. If any messages are printed starting with an absolute file name followed
by a colon, they are automatically hyperlinked; line numbers, column numbers,
line/column ranges, and trailing messages are supported here.
(Java stack traces are also hyperlinked if the corresponding source can be
found.) Modules are permitted to customize hyperlinking and other things using AntLogger.
You can thus
navigate typical build errors using F12 without leaving the source editor
window. If you need to halt the build prematurely, you may do so using Build | Stop Building.
Otherwise the build runs to completion, or failure. You
can run multiple builds at once. You can also adjust the verbosity level of Ant
to suppress noncritical information, or on the other hand to include debugging
information such as which commands are being run with what arguments at each
stage of the build.
The NetBeans IDE bundles some special Ant tasks which can only be run in the same VM as
the IDE, as they do something to the IDE itself. A build-impl.xml
may use these tasks in some targets—where appropriate, targets not using
them are also available in case you are running the script outside the IDE.
<nbbrowse> opens the IDE’s configured web browser on an
HTML file and may be used to e.g. display Javadoc after it is generated.
<nbjpdastart> prepares the IDE’s JPDA debugger to work
with a debugged program. Since these are run like any other Ant task, you can
use them in build.xml if you wish.
Targets in build-impl.xml normally reflect appropriate
dependencies: for example, a project is automatically built before it is run.
Ant tasks normally do internal dependency analysis, so e.g. Java source files
will not be recompiled gratuitously. A project will typically have a
Clean action that deletes all build products, and a Rebuild action
that first cleans and then does a regular build—plain Build is an
incremental build.
You can create multiple projects, and have multiple projects open in the GUI at
once. Currently you are free to open as many projects as you like at once
regardless of whether they are related to one another. Remember that little
depends on which projects are open; this is primarily a list of what projects
you would like to look at in the GUI.
A project may have some required projects, or informally subprojects.
(Two projects may both depend on the same subproject, but you cannot
have cycles.) Typically building a project first builds its subprojects
(recursively), and cleaning a project also cleans its subprojects—you can
disable this behavior in the GUI, however. Currently the GUI prompts you to open
subprojects when you open the parent project by default.
You do not need to do anything special to set dependency relations between
projects. Just use a build product of one project in another project, using the
GUI. For example, if in a Java project you configure the classpath to include a
JAR built by another project, you will have set up a dependency between the two
projects. If you remove that JAR from the classpath, you remove the dependency.
There is no special GUI for configuring build order—subprojects are simply
built before projects that depend on them.
The parent project keeps a reference to the location of the subproject on disk.
Using a VCS system, this will normally be a relative file path. See below for more information.
If you edit build.xml by hand, you can of course arrange to build
other projects (or any Ant scripts) as part of your build, using the
<ant> or <subant> tasks. Note that a
build-impl.xml, when building a foreign project, calls its
build.xml (rather than skipping to its
build-impl.xml), so you can freely mix a hand-customized project
with IDE-customized projects.
It is permitted for one project A to depend on projects B and C, both of which
in turn depend on a project D—i.e. for the dependency graph to be a DAG
(directed acyclic graph). This may happen if a generic library is used by
several components of an application. Currently the Ant build scripts will run
D’s build script twice if you ask to build A. This is probably not a
serious problem, as the incremental nature of builds means that the second run
will just complete quickly and quietly once Ant sees that everything in D is
already up to date, but it does add some overhead to the build (especially in a
deeply tangled DAG). This may be resolved in a future release.
The J2SE project type supports general Java applications, typically client-side.
It can contain one tree of Java sources in the src/ folder, some
JUnit tests in the test/ folder, and produces a single JAR file
from compiled classes and resources. Important targets include:
- Build
-
Compile Java sources and create the JAR. Clean and Rebuild are
as you would expect.
- Run
-
Run the program using a main(String[]) class.
- Javadoc
-
Create Javadoc for classes in the project.
- Test
-
Run JUnit tests. You can also run a single test case by itself.
- Debug
-
Debug the program, i.e. run it using a main(String[]) class but
attach the debugger to it. You can also debug a single test case if it is
failing but you do not know why.
Since vanilla J2SE development does not imply a particular way of packaging or
deploying an application, a J2SE project has no special distributable beyond the
JAR file. (Of course you can edit build.xml to do something more.)
A project type for a more structured platform such as Java WebStart or J2EE
could know exactly how to package the complete application, including references
to libraries produced by subprojects.
To build a project from the command line, or from another IDE, just run its Ant
script.
The project type may use Ant optional tasks which require specific libraries to
be in Ant’s classpath. For example, the plain J2SE project type supports
JUnit-based testing. Ant ships with the <junit> task, but
requires that junit.jar be present in Ant’s own classpath.
(Ant does not currently permit you to select the location of
junit.jar from the build script.) Therefore in order to run J2SE
projects from the command line, you need to include junit.jar in
$ANT_HOME/lib/, or your $CLASSPATH, or in a directory
specified by -lib, or in a path appended in ~/.antrc,
&c. The copy of junit.jar included in the IDE distribution will
work, but the user could use a different copy instead.
If you build a project from outside the IDE that needs some resources from
build.properties in the user directory, by default the build will
fail since only Ant running inside the IDE knows where to find this file. You
can either pass the location of build.properties to Ant using the
user.properties.file property, or point to a separately maintained
properties file (useful for SCM—see below). Or you can simply define
whatever properties you need in any other way supported by Ant.
In some cases the IDE may ship with libraries or other resources that are needed
by some project type, in which build.properties will automatically
include references to the library for use inside the IDE. When running a build
from outside the IDE, you can choose to refer to the IDE’s copy of the
library; or you may prefer to refer to your own copy stored elsewhere. (Library
paths should be individually overridable.) Strict software configuration
management (SCM) principles dictate that everything needed to build a
project should be kept in a version control system, ready for use by an
unattended headless build server with a tightly controlled environment. In such
a case, you would copy libraries shipped with the IDE into your VCS (or use your
own preferred copies) and adjust properties to refer to your preferred versions.
The GUI does not currently assist you in making this conversion to
a strict SCM style, though a future release might.
If a project’s build-impl.xml requires tasks that are shipped
with the IDE and not present in a standard Ant distribution—this is
not the case for a plain J2SE project, but other project types (e.g. for J2EE) need
to do this—the tasks will need to be defined in the build script according
to an overridable classpath, very much like any library definition. This should
be compatible with an SCM development style.
The primary code involved in the build system is divided into three portions: a
generic project system; an Ant-based implementation of that project system; and
some Java-specific facilities.
This section will just describe the concepts beyond the code architecture, not
every class and method. Reasonably complete Javadoc is available if you want to
know the details.
The projects/queries module defines some generic queries.
The term “query” is used to refer to a design pattern by which two
modules can communicate about a very concrete issue without needing to have a
hard dependency—or indeed without needing to know anything about the model
or design which decides the question. This design pattern is very important for
the build system since it permits gratuitous inter-module dependencies to be
avoided, and would permit very different architectures to be substituted for
various subsystems.
For example, the editor module should not have to comprehend a
complete model of how Java projects manage their sources, just in order to
provide appropriate code completion for a Java source. All it really
needs to know is the intended compile-time classpath for a given source
file—all other details are no interest in this context. Therefore, it
calls a query which maps a file to a classpath, and uses the result. The query
is implemented in this case by a Java-based project type which knows what the
classpath should be for a given source root. If the architecture of the project
type is radically changed, there will be no effect on the editor—in fact a
module could provide code completion information for certain sources not in
any project, if that were useful. The communication between the querier
and the provider is managed in a simple way by Lookup.
The queries in this module pertain to information about files and do not imply
the presence of any projects, though they are of especial use in conjunction
with projects. Other modules can and do define other query interfaces, in
addition to implementing them; generally a query interface should live in the
most general API-providing module that it can, so as to minimize gratuitous
module dependencies.
The projects/projectapi module defines the generic infrastructure
for projects. This infrastructure has no dependencies on Ant or Java; it could
in principle be used for other build systems (e.g. Maven), or other languages
(e.g. C++).
The Project interface represents one project, as loaded into memory
from disk. ProjectManager can be used to load and save projects.
“Loading” a project does not imply that it is in any way displayed
in the GUI (though projects displayed in the GUI will always be loaded); a
project is loaded quietly when someone first requests it. A project is always
kept in memory while it is modified; if it is not modified, it may be released
by the garbage collector. A ProjectFactory is responsible for
recognizing projects of a certain type on disk, loading them, and saving them
when they are modified. (For Ant-based projects, you do not need to
create your own ProjectFactory; see below.)
A project has an associated Lookup giving optional capabilities.
SubprojectProvider is a predefined capability of projects which
might have some “subprojects” (the meaning is intentionally left
vague, but see above for the typical interpretation).
ActionProvider is a capability of projects which can run some user
actions. ExtensibleMetadataProvider permits arbitrary modules to
store some information in a project without explicit support from the project
type—this may serve as a useful alternative to .nbattrs files
for many purposes.
FileOwnerQuery lets you find the project that owns a file. The
default implementation just searches for the nearest containing project
directory.
The projects/projectui module defines how projects may be displayed
in the GUI, and provides the standard generic UI elements.
LogicalViewProvider lets a project define a logical view for use in
the Projects tab, instead of just showing the raw folder as it appears
in the Files tab. This view might, for example, display Java
sources arranged in a flat package list, or EJBs—the possibilities are endless. (But some
voluntary UI consistency between project types is necessary to avoid a
hodgepodge appearance.) CustomizerProvider lets you provide a rich
GUI for customizing the project’s configuration; there are some support
SPIs for this.
The ant/project module contains mostly support classes which make
it straightforward to define an Ant-based project with the conventional
semantics described above. An AntBasedProjectType implementation is
registered to define one Ant-based project type, for example plain J2SE
projects. The project implementation receives an AntProjectHelper
instance which supplies a great deal of standardized behavior and which would
normally be used to implement Project methods and various optional
capabilities.
Design note: generally a project type should live in its own
module and nothing else should depend on it—it should not expose
any public packages etc. All of its functionality that is not directly
encapsulated in the project type itself should be exposed only indirectly
through query implementations. This design constraint ensures that it is
possible to produce additional project types with some of the same
functionality, without hacking other modules.
AntProjectHelper principally deals with reading and writing
project.xml (and private.xml), and
project.properties (and private.properties). It tracks
the modification status of the project automatically and handles persistence to
disk of these files. It also helps regenerate build-impl.xml from
project.xml with an XSLT stylesheet; implement GUI actions by
running Ant targets; interpret build properties as Ant itself would interpret
them, to make design-time features in the IDE match the build behavior as
closely as possible; and more.
You may also create a ReferenceHelper object which manages
subproject references according to the conventional semantics. Associated APIs
such as AntArtifact let an Ant-based project export a list of build
products it expects other projects might wish to use, or conversely let an
Ant-based project find appropriate build products from potential subprojects.
The java/api module defines some general APIs particular to the
Java language (or VM). Most significantly, this includes the Classpath API which
lets you both find and declare the classpath that will be used to compile a Java
source; the classpath that will be used to run it; etc.
java/project defines some SPIs helpful for implementors of
Java-oriented projects and some miscellaneous APIs.
The java/platform module defines an API for finding or registering
Java platforms—i.e. installations of the JRE or SDK. It supports multiple
editions such as J2SE, J2ME, and J2EE, as well as profiles such as are used
heavily by J2ME. java/j2seplatform provides an implementation of
the API for J2SE, and includes a wizard to autodetect J2SDK installations.
Portions of the platform information relevant to Ant are made available in
build.properties in the user directory.
The projects/libraries module lets you
register and search libraries, such as JARs that might be shared by multiple
projects. This will be used by project types to let you configure the location
of everything of interest about a library (e.g. binary, source, and Javadoc) in
one place in the GUI, and then quickly add a defined library to a project.
(Modules are also be able to declare information about libraries they bundle.)
Portions of the library information relevant to Ant are made available in
build.properties in the user directory.
java/j2seproject is the implementation of the J2SE project type.
Most of the code is currently concerned with providing an appropriate GUI,
especially for the project customizer. It also implements a number of queries,
and of course supplies the Ant-based project type itself.
While the build system does not directly implement any VCS-specific UI, it is
designed to be “VCS friendly”, i.e. to store metadata in such a way
as to make typical VCS operations work smoothly.
One aspect of VCS friendliness pertains to subproject references. Often the
subproject will reside in the same VCS repository as the parent project. If the
IDE determines that both projects are collocated in this way, it will use a
relative file path to the subproject in the parent project’s
configuration, so that the set of projects can be checked out from VCS as a unit
and be ready to run. Such relative paths are kept in the shared
project.properties. If the projects are not collocated in a
recognized way, an absolute file path will be used, in
private.properties.
The same consideration applies when deciding whether to use a relative or
absolute path for a plain file reference from a project, e.g. a JAR (not
included in a “library”) added to a Java project’s classpath.
The generic query CollocationQuery permits a VCS filesystem
implementation to tell other modules in the IDE whether files are collocated.
There is one implementation of this query in VCS module.
As noted above, projects should clearly separate
subdirectories which should and should not be kept in VCS. Typically everything
is versionable except for build output directories and
nbproject/private/. The generic
query SharabilityQuery permits a VCS to find out whether a given
file or directory ought to be considered versionable, without knowing any
details of the project system. VCS filesystem implementations
in NetBeans can check this query and use it to decide
whether or not to add a file or directory to the repository by default.
Note that
SharabilityQuery supports VCS implementations, such as ClearCase,
which need to know whether a directory will be shared before it is
created on disk.
A critical aspect of VCS friendliness is that all metadata files which are
written to by some GUI element in the IDE should be stored in a way that will
minimize the size of standard textual diffs, in order to reduce the chance of
merge conflicts, make code reviews more pleasant, etc. For typical Ant-based
projects this boils down to project.xml and
project.properties.
project.xml is formatted as a namespaced XML document. It is always
written to disk using conventional formatting (i.e. line breaks between elements
and four-space indentation). It consists of a short general section defined by
AntProjectHelper; a section controlled by the project type
provider, otherwise freeform; and zero or more sections supplied by other
modules using the ExtensibleMetadataProvider implementation
provided by AntProjectHelper, including a subproject references
section defined by ReferenceHelper. The stable sort order is
guaranteed for elements
in the general section; the root elements of the project type provider’s
section and each of the extended sections; and the subelements defined by any
code in the ant/project and java/j2seproject modules.
Other project types should also take care to ensure that their subelements are
stably sorted and otherwise produce minimal diffs when modified. (Also it is
expected that changes to project.xml are not requested unless some
actual change to the XML DOM tree was made, though this is not currently enforced.) Since
project.xml is not expected to be edited as text by typical users,
preservation of formatting (comments, whitespace, etc.) is not a high priority.
Similarly, project.properties is formatted as a Java properties
file. The EditableProperties class is designed for
reading/writting of properties. It is similar to java.util.Properties,
but it is designed to retain additional information needed for safe
hand-editing. Added functionality provided by this class compared to
java.util.Properties:
- minimizes textual diffs of changes, by preserving comments and
formatting in lines not affected by a change
- order of entries preserved during modifications whenever possible
- can associate comments with particular entries
- inserts new properties in lexicographic order
- can split long composite property values (e.g. classpaths) into multiple
lines that can be more easily diffed and merged
Unlike project.xml, project.properties is intended to
be editable as text by semiadvanced users, so preservation of formatting is
important.
If necessary to reduce the likelihood of merge conflicts in
project.xml, a couple of lines of whitespace or comments could be
introduced between sections. Typical merge tools have a “horizon” of
around three lines of text that they consider to indicate conflicting changes,
so this trick would prevent conflicts in case two different sections were both
modified. A similar trick could be used for project.properties,
though the syntactic simplicity of properties files makes merge conflicts less
onerous (and the relatively large number of independent properties makes them
less likely).
A project’s build-impl.xml file is kept under version
control, yet is regenerated from project.xml. In principle a merge
conflict could arise if two developers both modified a shared
project.xml and build-impl.xml was correspondingly
modified for each. In such a case, the developer responsible for resolving the
merge conflict could simply resolve any conflicts in project.xml,
delete build-impl.xml, and open the project in the IDE to force the
script to be regenerated.
build.xml is technically similar to build-impl.xml,
but is intended to be hand-editable, in which case there are no special
requirements for versioning—users who wish to avoid extraneous diffs can
do so themselves. If the user never makes any edits except through the GUI, then
it is automatically and quietly regenerated just like
build-impl.xml, in which case the same merge conflict situation
described above could be resolved in the same way.
There is no special code in the build system for handling VCS working directories.
MasterFS (openide/masterfs) is used which
makes all files on disk available to the
Filesystems API and guarantees a one-to-one mapping between files and file
objects.
The Versioning Manager independently handles configuration of VCS information.
Currently the build system provides no special GUI elements for interacting with
VCS modules. If a file in a project is in a VCS, the regular VCS context menu
should be available. Project type providers are also free to supply VCS-specific
annotations or actions in their logical views. Beyond that, no special support
is provided. However tighter integration in the UI between
projects and the VCS is planned.
The user is free to write top-level “starter” build scripts that
check out all sources for a project and its subprojects from a VCS before
building them, using standard Ant tasks. No special GUI support for this is
planned.
Currently there are no special provisions for ensuring that project metadata on
disk are valid before loading the project (e.g. in case a user made mistakes
when resolving a VCS merge conflict), or for ensuring that it is valid before
saving it (as insurance against buggy modules). W3C XML schemas do exist for all
the namespaces used in the infrastructure and standard project types, which can be
used to manually validate XML-formatted metadata (Alt-Shift-F9), but they are not currently
used at runtime. Developers of new project types are urged to develop schemas to
describe the syntax of their XML fragments.
(W3C XML Schema is much superior to DTDs for this purpose, not only because of
support for XML namespaces, but also because of support for associating multiple
schemas with a single document and validating fragments piecemeal. It is well
supported by Xerces.)
Regarding properties files, it is important to note that when you check out a
project from a VCS, private.properties will not yet exist. In this
case, if any properties are missing the opening of the project will warn the user and
ask him or her to resolve these problems using a provided GUI. This
UI can involve setting the paths to foreign projects or foreign
files, creating missing platforms or libraries, etc.
private.xml does not pose
any issues if it is missing—Ant does not look for it, and
AntProjectHelper automatically creates it if something needs to be
added to it.
Users with existing Ant build systems of substantial complexity, or those who
simply need advanced control over their build, are not likely to find the
IDE-generated build-impl.xmls adequate; and they may have special
requirements for projects that are not satisfied by stock project types, such as
multiple compilation units in a single project. The build system provides a
technique for such users to gain significant benefits of IDE integration while
maintaining full control over their build scripts.
It is helpful to think of project types in the following order, with later
entries providing more power but requiring more setup:
-
Project types supplied by the IDE with no editing of the build script. The
user might not know how to use Ant (or want to know); or they might be an
experienced Ant user but simply have no special requirements for this project.
They can create a new project out of the box (or import a simple source tree,
depending on the particular project type), customize a couple of things in GUI
dialogs, and begin development. The project type should support commonly
needed configuration parameters, but no more—the GUI should be kept
simple and easy to learn, and the purpose of the project (e.g. “build a
web application, deploy and test it”) should be predetermined and
focussed.
-
The same as #1, but with customizations made to the build script. The user has
some knowledge of Ant and wants to either make some modifications to the build
process not supported by the GUI (e.g. excluding some files from the created
JAR); and/or wants to add extra targets for functionality not covered by the
GUI (e.g. running Checkstyle).
-
The freeform project type. This user is comfortable with Ant and does not need
to rely on the IDE to help him or her write a build script. The user may have
special structural requirements such as multiple source roots with
interdependencies, in addition to the kinds of customizations possible in #2.
There are many ways to configure the project type, though within
limits—e.g. all source roots must be explicitly declared.
It is important to note that labels like “J2SE application” and
“J2EE web application” apply to levels #1 and #2. A particular
freeform project could be used as a J2SE application or a J2EE web application
or a VoiceXML extension library or anything else; the IDE does not know or
care, though it may care about certain aspects of e.g. J2EE technology, so e.g.
JSP code completion can work correctly.
-
A custom-built project type. This is a power user able to write NetBeans
module extensions who wants to provide tight integration for a specialized
class of project, which might be used only in-house (for a large suite of
projects), or might be more generally applicable. At this level, almost
anything is possible. While the author of the project type module needs to be
rather sophisticated, the user of a project managed by it need not
be.
The solution to #3 is a special freeform project type which defines a
project.xml syntax but which does not create a
build-impl.xml and assumes that the user is wholly responsible for
editing some build.xml (or even multiple scripts); in fact the user
should be able to use an existing build script unmodified if it already provides
all the functionality he or she needs. The project supplies a GUI customizer which
is just an editor for the major functionality exposed by project.xml;
the user can also manually edit
project.xml and various properties files as
text. The user creates a project.xml giving sufficient
information about the structure of the project to permit the freeform project
type to implement important “queries” and delegate to Ant when
running actions, but no more.
The user can use a GUI wizard to create
project.xml and there is also a project properties dialog
permitting further GUI customizations. Not all features of the freeform project
type are exposed in the GUI, only the most commonly needed functionality;
additional customizations can be made by editing project.xml
manually as text. (In which case syntactic or semantic validation errors are
automatically displayed in the Output Window when the file is saved.)
What does the freeform project.xml contain in practice?
-
A display name.
-
A sequence of fixed property definitions (e.g. config.dir=etc)
and property file load declarations (e.g.
${config.dir}/build.properties) that are used by the build
script(s) and might be relevant to the IDE for configuring other entries in
project.xml. The intention is to mirror the most commonly used
syntax variants of Ant’s <property> task so that the
user can make various configuration changes in property files and have them
affect both the Ant build and the IDE configuration at the same time. The IDE
tracks changes to these property files and updates its display and
behavior accordingly.
-
A mapping from IDE internal action names (e.g. compile.single) to
Ant targets to implement them (e.g.
${basedir}/build.xml#compile-one-file). The user can
set the target names and optionally the build script to use in each case.
If a user does not configure an Ant target for a given action, it is simply
disabled for that project.
In the case of context-sensitive actions (like Compile File), the user
can also configure an Ant property name to use to pass the
selection (e.g. javac.includes) as well as a predefined format
(e.g. “comma-separated list of package-root-relative slash-separated
file names”). The IDE would then run the Ant target with the correct
definition (e.g.
-Djavac.includes=org/foo/Foo.java,org/foo/Bar.java).
A list of Ant targets that
should get menu items in the context menu of the project’s logical view,
for commonly needed actions.
Some actions like Debug intrinsically require NetBeans-specific Ant
tasks to be used in the build script, as they act on the live IDE; clearly
such targets would not be useful except inside the NetBeans IDE. Other than
that they behave no differently from other targets.
In the case of IDE-oriented actions (including both context-sensitive and IDE-only actions), the IDE can in
some cases automatically generate a plausible initial implementation of the Ant target (and associated
mapping) for you automatically, if you try to run the action without a binding. This is currently done for
some actions; the rest are expected to be handled in a future release.
-
A list of top-level source directories, defaulting to just be the project
directory. Each should have a display name. This satisfies the
SourceGroup contract for Sources.TYPE_GENERIC and
also means that Find Files, Scan To-Do Items, etc. would work
correctly even if there are external source roots.
Also a list of file locations with display names to show in the
Projects tab: each could be a single file, a plain folder, or a Java
package root (shown as a package list). In any event, the Files tab
always shows all top-level source directories as plain folders.
-
A list of Java compilation units (one or perhaps multiple package roots
apiece). Each package root should give a display name for use in the IDE (e.g.
New File wizard) and a location inside or outside of the project
(perhaps expressed using Ant properties). This satisfies the
SourceGroup contract for Sources.TYPE_JAVA.
Each compilation unit should have a compile-time classpath (usually expressed
using Ant properties), and maybe also a bootstrap classpath and runtime
classpath if necessary. This satisfies the ClassPathProvider
contract. It should have a Java source level (e.g. 1.4); this
satisfies the SourceLevelQueryImplementation contract. Together
these things permit code completion and refactoring operations to work
accurately, as well as many other less obvious Java IDE features.
A compilation unit may also have an associated list of binary targets (e.g.
build/classes/ or dist/app.jar) which correspond to
it. This satisifies the SourceForBinaryQueryImplementation
contract, and improves code completion and refactoring as well as making Ant
stack trace hyperlinking more reliable.
-
Other information about specialized source folders relevant to J2EE
web application development, e.g. location of any document roots, or tag
libraries used by JSPs in a docroot.
-
Other miscellaneous information which is useful to the IDE but not strictly
required for everyday usage may be added. These things would be optional for the
user to specify as well. For example:
-
A marker that a given source root contains unit tests, so that Open Unit Test can work.
-
A list of build products (e.g. JARs) that can be used by stock IDE project
types such as in the J2SE project type’s classpath customizer (Add
Project… button).
-
A list of subprojects to open with the main project.
Just setting up a few Ant targets and a Java compilation unit or two is
suffice to have a project that you could build, run, and use refactoring and
code completion on just as well as in an IDE-supplied project type. And the
result is of course versionable so your colleagues can check out the project
including project.xml and begin work right away. If you configure
enough stuff in project.xml, you can simulate nearly all the
functionality of the J2SE project type and perhaps other project types except
for the customizer dialogs.
Some experimental work has also been done which indicates that it may be
feasible to automatically scan an existing Ant build script and detect such
information automatically; perhaps even updating this information every time the
build script is changed. Effectively it would be a freeform project but would
write or update much of your project.xml automatically. These
possibilities provide rich opportunities for future releases to improve the
workflow of developers working on large and complex applications.
There is also a special project type that supports NetBeans module development.
This
project type is tightly tuned to the special requirements of NetBeans module
development.
What happens to a user project when some part of the IDE is upgraded? Some infrastructure is currently in place
to handle such scenarios, though it may need to be expanded. Policies are still in development.
-
If a new version of a project type provider includes an updated XSLT stylesheet for
build-impl.xml, user projects are currently quietly upgraded when the project is opened. This
may be wrong.
-
If a new version of a project type provider makes use of a different set of standard Ant property names, a
user project’s project.properties can be quietly upgraded to include the new property
names and remove the obsoleted ones, when the project is opened.
-
Changes in the supported syntax of project.xml are possible between releases; if the user makes
configuration changes in the properties dialog which would require the new format, he or she is warned about
the format change and the project is written in the new format.
-
If the set of supported build targets for a project type changes with a new release, there is not much to do
other than warn the user if they have any build.xml customizations.
Though these problems are somewhat similar to questions about what to do with
old settings in the user directory, typically project configuration is much more
important to a user than unversioned, global per-user IDE settings, and project
metadata should be treated as precious data.
GeneratedFilesHelper permits you to determine whether
either of build.xml and build-impl.xml have been
modified manually by the user; whether they were generated from an older
project.xml; and whether they were generated from an older (or just
different) XSLT stylesheet. The J2SE project type regenerates both
build.xml and build-impl.xml when it is safe to do
so—i.e. when there have been no user modifications to the file. This is
done both when saving changes to project.xml, and when the project
is opened in the GUI with out-of-date scripts. You are warned if build-impl.xml
needs to be regenerated but has been edited.
Project type providers are currently left to their own devices as regards
upgrading domain-specific configuration data in project.xml. The
support SPIs permit various kinds of upgrade policies to be implemented.
Currently the project system expects that the project type is
atomic, i.e. a single project has a predefined set of behaviors. A
future release might expand this system to permit additional natures
or sets of behaviors to be plugged into the project, either when it is created
or perhaps after the fact. For example, a plain J2SE project could then be
extended to also support obfuscation of the JAR. Currently, this would be
possible with full GUI support only if obfuscation were directly supported by
the IDE module supplying the standard J2SE project type, though a user could
still manually configure the project to do anything Ant can do.
This feature is not currently planned due to its potential API complexity.
Instead, a reasonable set of capabilities are hardcoded into the supplied
project types, and users needing to do other things can edit
build.xml to support that. As the J2SE project type demonstrates,
supporting a variety of build steps in a project type need not be difficult,
especially if Ant tasks to support the feature already exist.
If a third-party module really needs to extend a project with a new capability,
as a workaround it could insert added targets into build.xml
itself, perhaps using a wizard. Or it could simply define non-Ant-based actions
in the context menu of object types it defines—for example, the XML
modules do this for XML validation and XSLT processing, without project system participation.
For a future release the Ant-based project type infrastructure could
provide an option for the project type to declare its extensibility by natures,
for which an SPI would be created. A nature might need to be able to influence
the initial build.xml; the generated build-impl.xml;
the initially generated properties files; the customizer dialog; the logical
view; the project’s Lookup; etc. Hopefully such an API
extension can be done compatibly or at least with only minor incompatibilities.
However no concrete proposal has been put on the table to date.
Update for NetBeans 6.0: see Project Groups
Since a number of people have asked why the build system does not define the
concept of a “portfolio”, it is worth looking at the question in
detail. The main answer is that there are many possible interpretations of the
question, and under some interpretations, the requirement is already satisfied.
Let us enumerate some possible meanings of the term first, with some more
precise synonyms:
- master project
-
A project which has build-time dependencies on some subprojects, perhaps
recursively. In order to build the master project, you must necessarily build
the subprojects. Often the build products of the subprojects are not just used
during the master project’s build, but are incorporated into the final
distributable.
There are numerous examples of this pattern. For example, a J2EE EAR can be
considered a master project with subprojects which are WARs, EJB-JARs, or
provisionable client JARs. A Java WebStart application can contain various
libraries which are referenced from the JNLP file. A J2ME MIDlet can inline
libraries into its JAR as it is being packaged and verified. For plain J2SE
projects, the build-time dependency is clear, though there is no standard way
of packaging up the subprojects.
Master projects are directly supported by the build system’s
infrastructure and implemented for J2SE projects. They are currently loosely
supported in the GUI: opening a master project by default opens its
subprojects too.
The project system GUI currently lets you configure the master project open at
any given time. (The GUI label is Main Project.) Some project actions,
such as Run, apply to it (when invoked from the global menu or
toolbar).
- aggregate project
-
A pseudo-project which just aggregates some number of subprojects together in
a group. They need not have any particular relationship to one another, with
respect to the build process or otherwise; but it may be desirable to set an
explicit relative build order of the subprojects in the aggregate project. The
aggregate project might package up each of the subprojects into one big
distributable (somehow). It might define one or perhaps multiple “entry
points” (subprojects) for running, deployment, etc. It might have
“batch” targets to build documentation for each subproject in
turn, run all unit tests, etc.
Currently aggregate projects are not directly supported by the build system,
but it would be reasonably straightforward to create an aggregate project type
using the existing APIs—perhaps a generic aggregate project type
applicable to many application domains, or perhaps a more specific type.
Aggregate projects would work much the same as master projects as far as the
UI is concerned; the differences mainly lie in build semantics.
Of course any user comfortable with Ant can create a wrapper build script with
this kind of behavior; the integration in the IDE’s UI will not be
tight, however.
- open project list
-
A simple list of projects that should be opened (or closed) as a unit. They
may have no further relationship to one another; the user simply wishes to see
them all at the same time.
The current build system UI does not support this concept, though it could
easily be extended to do so. You need to open and close the desired projects
individually. However you are permitted to have any projects open at once that
you like.
- VCS project group
-
A group of projects all collocated in a VCS—perhaps as sibling
directories, perhaps as one top directory with subdirectories. No particular
build-time semantics or interdependencies are implied by this term, so it can
be used to qualify other terms like “master project” or
“open project list”. The UI would ideally make it easy to check
out the group as a whole and open it all automatically—probably by
making an open project list as a text file shared in the VCS and using
relative file paths.
The build system does not directly support this concept, though it would be
straightforward to add to the UI infrastructure.
- scope for refactoring (…)
-
The set of projects open in the GUI is
the scope for large operations such as Java refactoring.
I.e. only sources contained in the scope are considered for updates when
renaming a method, etc.
A note: since the scope for refactoring is limited in this way,
the refactoring operation may break the build
of some project which was accidentally omitted from the list.
Currently the IDE makes no attempt to warn the user about this condition,
though it conceivably could (e.g. using
a reverse dependency index).
Currently the build system provides no special support for this concept beyond
the list of projects open in the GUI.
Some other IDE functionality need a similar scope: for example, the Java
Fast Open feature needs to locate a source file by fully-qualified
name. There might be multiple such sources in projects on disk, especially if
the user has been keeping checkouts of multiple VCS branches of an
application. Therefore the action searches only projects open in the GUI
for a matching source.
(Beware of overgeneralizing the “scope” concept to actions to
which there is already a more clearly-defined scope. For example, the
apparently similar Go To Class (or method, etc.) context menu item and
keyboard shortcut in the Java source editor should actually behave
differently. In this case, the scope in which the IDE needs to find the named
class is clearly defined by the logic of the situation: it should be the
sourcepath of the selected Java source file. I.e. the IDE should always jump
to the source against which this file would actually be compiled, regardless
of what is open in the GUI. The distinction may sound subtle, but it becomes
very important when you are rapidly switching back and forth between several
VCS branches: if A.java compiles against B.java, and
you happen to have a copy of A.java checked out from the
release44 branch open in the editor and the caret is over a
method defined in B.java and you press Alt-G, you
certainly wish to go to the release44 version of
B.java. You could become quite confused if you were sent to a
checkout of B.java from a different branch, which might not even
contain that method!)
Clearly there are many potential interpretations of the term
“portfolio”, often partially overlapping. Currently a more
sophisticated portfolio system is not planned simply because it
will require a lot of time to do the usability studies, surveys, and analyses
necessary to judge which of these concepts are valuable, intuitive, and
expected.
Various sorts of portfolios can be added for a future release if there is a
broad consensus on what functionality should be subsumed. In the meantime, the
master project concept is likely to be adequate for the common application
types: plain J2SE application; J2ME MIDlet; J2EE WAR; J2EE EAR.
Most existing Ant-based applications with nested components use either the
master or aggregate project patterns, so this style should come as no surprise
to a developer accustomed to working on large projects with a scripted build
system. Typically subprojects are physically contained within the master
project’s directory, which effectively makes them VCS project groups as
well.
(An interesting side note is provided by the open-source Maven project comprehension system, for
which there is also a project type provider. While
Maven-based projects typically follow the master project pattern, the
subprojects need not form a VCS project group as such: instead you can download
a particular binary release of a foreign project as part of your build. Systems
such as Apache Gump make it easy
to ensure that open-source prerequisites are stable and available.)
Sometimes a development team will write a wrapper script (see above) that checks out a configured list of VCS modules,
sometimes from different branches; this can be considered a combination of a VCS
project group and an open project list.
