DivByZero.com

Much ado about scripting, Linux & Eclipse: card subject to change

2012-12-07

The JBoss Developers' Song

Hey! Guess what?

JBoss Tools 4.0 and JBoss Developer Studio 6.0 are available today. So... a quick tune in tribute to all the hard-working people who made it happen.

Who squashes bugs users have found?
Who keeps the unresolved count down?
We do! We do!

Who answers to the Will of Max?

Who deals with all of the PEBCAKs?

We do! We do!

Who closed/resolved a thousand bugs?

Who attends all of those JBUGs?

We do! We do!

Who fills more roles than Tom Hanks?

Who gets the top Marketplace ranks? 1

We do! We do!


1 - As of 2012/12/07, JBoss Tools + JBoss Developer Studio successful Eclipse Marketplace installs for Helios, Indigo, and Juno total over 141,000. SpringIDE and Spring Tool Suite installs total over 163,000. This puts us #5 behind only Maven, Subclipse, Subversive, and Spring.

2012-06-19

Open Source is Painless (Theme from JBDS)

With JBDS 5.0.0.CR1 and JBoss Tools 3.3.0.CR1 out, and GA/Final releases just around the corner, it's high time for a new song.

Hell's bells, it's been over 3 years since my last music-related post.

The idea for doing this song came after spending a day creating a diagram showing how JBoss Tools and Developer Studio are built, followed by myarboro's reaction to the complexity.

Diagram by Dia

Through early morning mail I see
Visions of JIRAs for me
Build jobs red unexpectedly
I realize and I can see...

That Open Source is painless
Even with non-stop changes
And yet we give all this away for free

The game of rel-eng's hard to play
We struggle through it anyway
A business card I'll never lay
Deprecated: it's LinkedIn's day

Jenkins builds are painless
They track so many changes
And ever onward moves complexity

The use of Git trumps SVN
But migrating's hard to begin
When your code base is not so slim
The pain grows stronger... bear & grin

Using github's painless
Pull/push requests for changes
And I can take or leave it if I please

A Turing test once asked of me
To answer questions that are key
'To use Tycho or PDE?'
And I replied 'Not PDE!'

Releng'ing is painless
It is a game of changes
And still we give all this away for free
And you can do the same thing if you please...

Johnny Mandel & Mike Altman - Suicide is Painless (Theme from M.A.S.H.)

2012-06-06

Managing Jenkins job configurations

In JBoss Tools and Developer Studio, we manage a lot of build jobs in Jenkins. In fact, for the 3.2.x/4.x and 3.3.x/5.x streams, there are over 195 jobs. When we start building our next year's first milestone, we'll spawn another 40+ jobs.

Here are some of them:

To assist in performance, we use maven profiles in our parent pom to allow data to be shared outside the slaves' workspaces, without using shared workspaces (as that can lead to conflicts when multiple maven processes try to write to the same .m2 repo). Here's an example:

  <!-- same contents as jbosstools-nightly-staging-composite, but locally 
   available (to improve network lag) -->
  <profile>
   <id>local.composite</id>
   <activation>
    <activeByDefault>false</activeByDefault>
   </activation>
   <repositories>
    <repository>
     <id>local.composite</id>
     <url>${local.composite}</url>
     <layout>p2</layout>
     <snapshots>
      <enabled>true</enabled>
     </snapshots>
     <releases>
      <enabled>true</enabled>
     </releases>
    </repository>
   </repositories>
  </profile>

We also use profiles to turn on code coverage analysis or to take advantage of Jenkins variables like BUILD_NUMBER when setting a timestamped qualifier for features & plugins:

  <profile>
   <id>hudson</id>
   <activation>
    <property>
     <name>BUILD_NUMBER</name>
    </property>
   </activation>
   <properties>
    <local.site>file:///home/hudson/static_build_env/jbds/target-platform_3.3.indigo.SR2/e372M-wtp332M.target/</local.site>
   </properties>
   <build>
    <plugins>
     <plugin>
      <groupId>org.eclipse.tycho</groupId>
      <artifactId>tycho-packaging-plugin</artifactId>
      <version>${tychoVersion}</version>
      <configuration>
       <format>'v'yyyyMMdd-HHmm'-H${BUILD_NUMBER}-${BUILD_ALIAS}'</format>
       <archiveSite>true</archiveSite>
      </configuration>
     </plugin>
    </plugins>
   </build>
  </profile>

But how do you deal with hundreds of jobs' config files, and how do you update them all easily w/o hours of in-browser clicking?

We maintain the job config files (config.xml) offline.

To do this, we use a maven plugin I wrote to fetch jobs matching a given view & regular expression and store them locally on disk using the same structure as on the server.

Then that same plugin can be used to push the config.xml files BACK to the server after making changes to one (or all) the files.

For an extra level of auditing, we also commit these locally cached config.xml files to SVN so we can track their history. Admittedly Jenkins provides this functionality natively, but when you're POSTing changes to config.xml files the server doesn't always notice a change and record the delta, so having a backup (particularly one you can diff offline) is never a bad idea.

2011-12-11

Build Nomenclature Conventions: What's in a name?

The following post is inspired by Mickael Istria's recent blog, Call a spade a spade, and a Nightly a Snapshot.

When I was doing builds for the Eclipse Modeling Project, I-builds were weekly published nightlies -- same level of stability as a SNAPSHOT (to use Maven parlance) or nightly, but published on a weekly schedule to bridge the gap between nightly/daily/SNAPSHOT/CI builds and the every-6-weeks milestone releases. The goal was to provide something stable enough for early adopters to grab once a week, but without the non-stop flux of nightlies. Regardless of the label on the build, the process was the same: tag CVS, then build using that tag.

The Final/GA/Release ("R") builds were done as simple renames of the last good milestone or release candidate build, so as to ensure binary-compatibility w/ the last-tested milestone/RC. The same was true for "M" and "S" builds -- they were just renamed "I" builds, and the letter was there simply to differentiate between a maintenance build (M), a stable milestone (S), or release (R).

Branching only happened when a release was done and it was time to produce the maintenance stream vs. the ongoing next-year-release. Sometimes branching would happen AFTER the x.y.1 maintenance because it saved duplication of commits in the x.y+1.0 and x.y.1 streams.

--

Now at JBoss, we publish "nightly" builds, which are keyed to SVN changes and therefore could be as often as hourly or as infrequent as weekly, depending on what's happening in the repo.

We also do milestone builds about once ever 6-8 weeks (similar to the Eclipse.org release train schedules), which is more carefully vetted, tested, and QE'd. It is produced using the same *process* as the nightlies, but are named differently and pulled from a freshly-created stable branch in the repo (so its degree of change/churn is less). (Branching happens right before every milestone or release candidate so that hardening/stabilization/documentation can happen in the branch while trunk stays open for new development.)

--

Bottom line -- I've only ever needed three types of builds, regardless of nomenclature or labelling differences. And of these 3, the last 2 are the same thing but renamed to underline the build quality/stability:

* nightly/CI/integration/weekly/SNAPSHOT build (unstable, for bleeding edge adopters)

* development milestone (probably a re-christened nightly; stable, early adopters)

* stable release / Final / GA (probably a re-christened milestone; release quality)

--

So... does it matter if it's called nightly, integration or SNAPSHOT? or Stable, Milestone, Maintenance, Final, GA or Release? As long as it's easily reproducible (yeah, Tycho!), what's in a name?

2011-11-09

HOWTO: Make KDE remember dual-monitor randr settings

Every time I boot up, KDE appears to forget that I want my monitors to be positioned left-to-right and instead defaults to mirrored config. But, after a lot of cursing and a little googling, I found an answer so it'll no so much keep your settings, but reset its broken config to your settings.

1. Hit ALT-F2, then enter "display" to run the Display Settings app.

2. Configure your settings as you'd like. Note that if the Apply button isn't active after your changes, you can change/revert something like a Position: button to make it active.

3. On restart, KDE may forget your dual-monitor settings. So, to prevent this, go look in your ~/.kde/share/config/krandrrc file:

[Display]
ApplyOnStartup=true
StartupCommands=xrandr --output "DVI-I-1" --pos 1920x0 --mode 1920x1200 --refresh 59.9502\nxrandr --output "HDMI-1" --pos 0x130 --mode 1920x1080 --refresh 60\nxrandr --noprimary

4. Copy the configuration into a new file, and replace \n with newlines. I like to put scripts like this in /etc/X11 because they relate to screen res and positioning.

# from ~/.kde/share/config/krandrrc
xrandr --output "DVI-I-1" --pos 1920x0 --mode 1920x1200 --refresh 59.9502 
xrandr --output "HDMI-1" --pos 0x130 --mode 1920x1080 --refresh 60 
xrandr --noprimary

5. Ensure the script is readable/executable for all users:

chmod 755 /etc/X11/1920x2.sh

6. Hit ALT-F2, then enter "autostart" to run the Autostart config tool.

7. Click Add script... and browse for the script you created above.

8. Reboot and watch the magic unfold.

2011-10-29

HOWTO: See what happened in SVN between builds

I was recently asked how to determine what changed between two builds. Jenkins provides nice interlinks into JIRA (issues), Fisheye (source changes), SVN (sources), but let's say you want to kick things a little more old school and investigate the old way... or the builds you want to compare are no longer shown in Jenkins because they expired and their metadata was automatically purged.

If you can't just look at the changelog in Jenkins to see what revision of source was used for the build, you can check the SVN log to find revision numbers based on the timestamp of the build.

So, if your build was generated on 2011-10-18, you can see that the log shows the last commit before that build was this:

$ svn log http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x/

...

r35735 | bfitzpat | 2011-10-17 15:35:23 -0400 (Mon, 17 Oct 2011) | 2 lines
Changed paths:
   A esb/plugins/.project
   M esb/plugins/org.jboss.tools.esb.project.core/src/org/jboss/tools/esb/core/runtime/ESBRuntimeResolver_410.java

JBDS-1889 - Now checking for juddi-client-3.1.2.jar as well as 3.1.0 and 3.1.1 when seeing if the runtime includes ESB 4.10

...

Want to see actual diffs between that build and the latest one?

$ svn diff http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x/@35735 http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x/

Or, if you want to collect just the section of log relevant to the change:

$ svn log -r35735:HEAD http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x/

Of course if you have all the sources locally, you don't need to log or diff via a URL - you can simply use local file paths. And if like me you use git-svn instead of pure svn, you can use that to diff or log too.

If you want to easily determine when a branch was created and get the SVN revision number for that branch point, use this:

# from r28571, returns -r28571:HEAD
rev=$(svn log --stop-on-copy \
  http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \
  | egrep "r[0-9]+" | tail -1 | sed -e "s#\(r[0-9]\+\).\+#-\1:HEAD#")

If you'd like to view a specific svn revision in your browser, use !svn/bc/REVISION_NUMBER/ before the branch and path to file or folder:

http://svn.jboss.org/repos/jbosstools/!svn/bc/35735/branches/jbosstools-3.2.x/

2011-10-26

HOWTO: Use Maven, Ant, and XSLT to scrub unwanted p2 metadata from an update site

Some time ago, I wrote about Using p2.inf to add/remove update sites. Tonight I found a simpler way to remove references in p2 metadata to external 3rd party sites.

For example, say you're repackaging some 3rd party features onto your own site, but don't want those features to provide references to the vendor's own update sites because you want to ensure that your product's site will only result in your sanctioned version being installed.

When you generate an update site, p2 pulls the information in the included features and will result in a section of references in the site's metadata that looks like this:

  <references size="6">
    <repository uri="http://download.eclipse.org/egit/updates" url="http://download.eclipse.org/egit/updates" type="0" options="0"/>
    <repository uri="http://subclipse.tigris.org/update_1.6.x" url="http://subclipse.tigris.org/update_1.6.x" type="1" options="0"/>
    <repository uri="http://download.eclipse.org/egit/updates" url="http://download.eclipse.org/egit/updates" type="1" options="0"/>
    <repository uri="http://subclipse.tigris.org/update_1.6.x" url="http://subclipse.tigris.org/update_1.6.x" type="0" options="0"/>
    <repository uri="http://eclipse.svnkit.com/1.3.x/" url="http://eclipse.svnkit.com/1.3.x/" type="0" options="0"/>
    <repository uri="http://eclipse.svnkit.com/1.3.x/" url="http://eclipse.svnkit.com/1.3.x/" type="1" options="0"/>
  </references>
To remove that, you can play with p2.inf directives, or you can simply perform an XSL transformation on the generated content.xml (inside content.jar, if your metadata is compressed) to remove the <references/> node:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
<xsl:template match="/">
        <xsl:apply-templates select="*"/>
</xsl:template>
<xsl:template match="*">
        <xsl:copy >
                <xsl:for-each select="@*">
                        <xsl:copy />
                </xsl:for-each>
                <xsl:apply-templates />
        </xsl:copy>
</xsl:template>
<xsl:template match="references" />
</xsl:stylesheet>
If you're generating your update site w/ Tycho, this transform can be called via a simple Ant script:

       <target name="remove.references">
                <!-- requires ant-contrib only if you like using if-then-else structures -->
                <if>
                        <available file="${update.site.source.dir}/content.jar" type="file" />
                        <then>
                                <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" />
                                <delete file="${update.site.source.dir}/content.jar" />
                        </then>
                </if>
                <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" />
                <xslt style="remove-references.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" />
                <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" />
                <delete file="${update.site.source.dir}/content.xml" />
                <delete file="${update.site.source.dir}/content.old.xml" />
        </target>

Then, in your site's pom.xml, to call the Ant script, do this:

       <build>
                <plugins>
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-antrun-plugin</artifactId>
                                <!-- make sure this variable is defined, eg., set to 1.3 -->
                                <version>${maven.antrun.plugin.version}</version>
                                <executions>
                                        <execution>
                                                <id>install</id>
                                                <phase>install</phase>
                                                <configuration>
                                                        <quiet>true</quiet>
                                                        <tasks>
                                                                <!-- called AFTER generating update site + zip to tweak content -->
                                                                <ant antfile="build.xml">
                                                                        <property name="SOME_ANT_VARIABLE" value="${SOME_MAVEN_VARIABLE}" />
                                                                </ant>
                                                        </tasks>
                                                </configuration>
                                                <goals>
                                                        <goal>run</goal>
                                                </goals>
                                        </execution>
                                </executions>
                                <dependencies>
                                        <!-- some dependencies your ant script might need -->
                                        <dependency>
                                                <groupId>commons-net</groupId>
                                                <artifactId>commons-net</artifactId>
                                                <version>1.4.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-nodeps</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-trax</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-commons-net</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-apache-regexp</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>ant-contrib</groupId>
                                                <artifactId>ant-contrib</artifactId>
                                                <version>1.0b3</version>
                                        </dependency>
                                </dependencies>
                        </plugin>
                </plugins>
        </build>

I suppose there's probably a way to call a transform directly from Maven w/o the Ant wrapper, but this allows unpacking and repacking of the content.jar to get at the content.xml file.

2011-09-01

HOWTO: Move around between desktops & windows with keyboard or mouse

Recently installed Fedora 15 KDE spin, partly because the XFCE spins wouldn't boot from CD but also because I've heard less-than-favourable things about Gnome3 and because I'm addicted to Konqueror as a graphical sftp/scp/ssh viewer, so figured might as well use kdm instead of xfwm4 or gdm.

Still having some problems getting my 1600x1200 (or 1920x1200) monitor to do anything more than 1024x768 on the VGA port of the video card (works fine on the DisplayPort connector, either directly or via a DP-to-DVI cable, but not on the VGA connector, even with xorg.conf hackery). That said the options for display/monitor management under KDE are much better than under XFCE, and this is the first time I've been able to get two monitors working without HOURS of hacking away at xorg.conf scripts. So... big props for this release *almost* Just Working.

Workaround I'm trying next is to install a second video card. Will update when/if that solves the problem once it arrives.

But video resolution aside, I did recently figure out how to set keyboard bindings for moving windows between desktops (thanks to David Fisco). From the K-menu, select System Settings > Shortcuts and Gestures > Global Keyboard Shortcuts > KDE Component: KWin > "Window One Desktop To The Left/Right":

... and for switching between desktops ("Switch To Next/Previous Desktop")

Also recently discovered some fun options for switching between windows (on all desktops). From the K-menu, select System Settings > Desktop Effects > Enable desktop effects > Effect for window switching: Present Windows (or any of the other options).

There's also System Settings > Window Behavior > Task Switcher > Effect: Present Windows:

You might want to set an animation for switching between desktops, though I find with multiple monitors this can be a bit dizzying. From the K-menu, select System Settings > Workspace Behavior > Virtual Desktops > Switching > Animation: Desktop Cube Animation. For something more subtle, try "Fade Desktop".

Finally, you may want to set screen edge behaviours, such as making Present Windows appear when you cursor to the top-center of your screen. System Settings > Workspace Behavior > Screen Edges > right-click a target zone:

2011-08-16

Mounting Linux LVM drives

Because my Thinkpad X200 has finally, after just under 3 years, decided to give up the ghost via a FAN ERROR and refusal to start (POST beeps & auto-shutdown), I'm now faced with the task of recovering all the data on the drive (about 120G) across multiple partitions.

Here's the drive layout, as per cfdisk:

                                   cfdisk (util-linux-ng 2.17.2)

                                        Disk Drive: /dev/sdb
                                 Size: 160041885696 bytes, 160.0 GB
                       Heads: 255   Sectors per Track: 63   Cylinders: 19457

     Name           Flags         Part Type    FS Type              [Label]            Size (MB)
 --------------------------------------------------------------------------------------------------
     sdb1                          Primary     NTFS                 [^B]                26214.44   *
     sdb2           Boot           Primary     Linux ext3                                 209.72   *
                                   Logical     Free Space                                   3.68   *
     sdb5                          Logical     Linux ext3           [HOME]             106043.70   *
     sdb6           NC             Logical     Linux LVM                                20970.48   *
                                   Logical     Free Space                                   1.09   *
     sdb4                          Primary     Compaq diagnostics                        6595.71   *
                                               Unusable                                     0.49   *

So, under a Fedora 13 LiveCD, the /boot (sdb2) and /home (sdb5) partitions automounted, along with the WinXP (sdb1) partition. But the root partition (/, part of sdb6) would not as it's part of a LVM. After a quick burst of googling, I found this solution, which digests down to simply this:

yum install lvm2 -y; # install support for lvm2
pvscan # scan vol groups
vgchange vg_x2lappy -a y # mark your vol group active
lvscan # scan for logical volumes 
mkdir /media/sdb6 # create a mount point
mount /dev/vg_x2lappy/lv_root /media/sdb6/ # mount the lv
cd /media/sdb6/; ls -la # take off every zig!

2011-07-27

MANIFEST.MF and feature.xml versioning rules

I'm forever forgetting what the rules are for dependency declarations in MANIFEST.MF and feature.xml for osgi plugins and features. And Googling often results in frustration rather than an answer. So, because today I actually found a concise list of the rules, I thought I'd repost them here, with some minor edits to help clarify.

OSGi Plugin Version Ranges

Dependencies on bundles and packages have an associated version range which is specified using an interval notation: a square bracket “[” or “]” denotes an inclusive end of the range and a round bracket “(” or “)” denotes an exclusive end of the range. Where one end of the range is to be included and the other excluded, it is permitted to pair a round bracket with a square bracket. The examples below make this clear.

If a single version number is used where a version range is required this does not indicate a single version, but the range starting from that version and including all higher versions.

There are four common cases:

  • A “strict” version range, such as [1.2.3,1.2.3], which denotes that version and only that version.

  • A “half-open” range, such as [1.2.3,2.0.0), which has an inclusive lower limit and an exclusive upper limit, denoting version 1.2.3 and any version after this, up to, but not including, version 2.0.0.

  • An “unbounded” version range, such as 1.2.3, which denotes version 1.2.3 and all later versions.

  • No version range, which denotes any version will be acceptable. NOT RECOMMENDED.

The complete text of the above snippet can be seen here (or here as PDF).

Example:

Require-Bundle: org.eclipse.core.runtime;bundle-version="[3.4.0,4.0.0)",
 org.eclipse.core.resources;bundle-version="[3.4.0,4.0.0)",
 org.eclipse.ui.ide;bundle-version="[3.4.0,4.0.0)",
 org.eclipse.ui.navigator;bundle-version="3.5.100",
 com.ibm.icu

See also:


In terms of feature manifest (feature.xml) rules, help.eclipse.org has pretty good documentation, but the most important thing to remember - and what I often have to look up - is how to state the matching rules for required upstream features & plugins. Experience says it's always better to state things explicitly so there's no downstream guesswork needed and anyone reading your manifest knows EXACTLY what version(s) are required for or compatible with your feature. Plus, while YOU might be using PDE UI to build, someone else might be using Tycho and Maven, and every tool can interpret missing metadata their own way.

When in doubt, spell it out.

Valid values and processing are as follows:
  • if version attribute is not specified, the match attribute (if specified) is ignored.
  • perfect - dependent plug-in version must match exactly the specified version. If "patch" is "true", "perfect" is assumed and other values cannot be set. [1.2.3,1.2.3]
  • equivalent - dependent plug-in version must be at least at the version specified, or at a higher service level (major and minor version levels must equal the specified version). [1.2.3,1.3)
  • compatible - dependent plug-in version must be at least at the version specified, or at a higher service level or minor level (major version level must equal the specified version). [1.2.3,2.0)
  • greaterOrEqual - dependent plug-in version must be at least at the version specified, or at a higher service, minor or major level. 1.2.3
The complete text of the above snippet can be seen here.

Example:

<requires>
  <import feature="org.eclipse.m2e.feature" version="1.0.0" match="compatible"/>
  <import feature="org.maven.ide.eclipse.wtp.feature" version="0.13.0" match="greaterOrEqual"/>

  <plugin id="ch.qos.logback.classic" version="0.9.27.v20110224-1110" match="greaterOrEqual"/>
  <plugin id="ch.qos.logback.core" version="0.9.27.v20110224-1110" match="greaterOrEqual"/>
  <plugin id="ch.qos.logback.slf4j" version="0.9.27.v20110224-1110" match="greaterOrEqual"/>
  <plugin id="org.slf4j.api" version="1.6.1.v20100831-0715" match="compatible"/>
  <plugin id="com.ning.async-http-client" version="1.6.3.201106061504" match="equivalent"/>
  <plugin id="org.jboss.netty" version="3.2.4.Final-201106061504" match="perfect"/>
  <plugin id="org.hamcrest.core" version="1.1.0.v20090501071000" match="equivalent"/>
</requires>