Planet Apache

Justin Mason: Links for 2013-06-18

2013-06-18T23:58:02+00:00

Open Rights Group – EU Commission caved to US demands to drop anti-PRISM privacy clause

Reports this week revealed that the US successfully pressed the European Commission to drop sections of the Data Protection Regulation that would, as the Financial Times explains, “have nullified any US request for technology and telecoms companies to hand over data on EU citizens. The article [...] would have prohibited transfers of personal information to a third country under a legal request, for example the one used by the NSA for their PRISM programme, unless “expressly authorized by an international agreement or provided for by mutual legal assistance treaties or approved by a supervisory authority.” The Article was deleted from the draft Regulation proper, which was published shortly afterwards in January 2012. The reports suggest this was due to intense pressure from the US. Commission Vice-President Viviane Reding favoured keeping the the clause, but other Commissioners seemingly did not grasp the significance of the article.

(tags: org privacy us surveillance fisaaa viviane-reding prism nsa ec eu data-protection)
Verified by Visa and MasterCard SecureCode kill 10-12% of your business

As Chris Shiflett noted: not only are they bad for security, they’re bad for business too.
12 percent of users consider abandoning [an online shopping transaction] when they see either the Verified by Visa or the American Express SafeKey logos, while 10 percent will consider abandoning when the see the MasterCard Secure card logo.

(tags: ecommerce vbv online-shopping mastercard visa securecode security fail)
The Cold Hard Facts of Freezing to Death

an amazing account of near-death from hypothermia (via Dor)

(tags: via:dor hypothermia cold medicine science non-fiction)

Dave Johnson: Tomorrow night: How Wayin does Cloud Orchestration

2013-06-18T22:06:23+00:00

I've already mentioned this on Twitter and LinkedIn, but just in case you missed it: I'll be speaking tomorrow night at the Triangle AWS and Triangle DevOps joint meetup at Argyle Social in Durham, NC. I'll give a quick overview of cloud orchestration and Wayin Hub. Then I'll dive into the details of how we automate deployment, scaling and backups for Wayin Hub using AWS and AWS Cloud Formation.

As a little teaser, here's a GIF animation of my automated deployment slide:

For more information check the Triangle DevOps page for the event.

Rich Bowen: RedHat Summit Summary

2013-06-18T20:05:37+00:00

Last week I attended the Red Hat Summit in Boston. It was, for me, equal parts pep rally and intensive OpenStack training.

Jim Whitehurst's keynote was just great, because it reemphasized how much RedHat really *gets* Open Source, at all levels of the organization. So, this part was pep rally for me, and confirmed to me that RedHat is the place where I want to be. Same for Paul Cormier's keynote. Both of these are well worth watching if you care about cloud computing, IaaS, or PaaS, or expect to at any time in the near future.

I attended a number of sessions about OpenStack, and you can see a wrapup of all of that content in Perry's blog post about the conference.

And I helped out at the RDO table in the Developers Lounge. In the process I met many of the engineers that I'll be working with, and I learned quite a bit about RDO and OpenStack, as well as who I need to go to when there's something I don't know yet. And I got to play around some with TryStack, a free service where you can experiment with an RDO installation, launch virtual machines, and connect in to them to see how RDO behaves.

There's a huge amount of interest in OpenStack, and the ecosystem around it is full of really cool stuff. I was particularly interested in OpenShift, with which you can launch a non-trivial webapp in just minutes minutes. Very cool stuff.

Another high point of the week was the RedHat Summit 5K.

There were a few hundred people in the race, which wasn't a traditional road race, in the sense that there wasn't any official time keeper, and traffic wasn't stopped. We had pace groups (I ran with the 8:30 minute group), and a pacer who knew the route. I had set a goal of breaking 27, and I ran a 25:32, with which I was very pleased. This was the first 5k I've run since, I believe, 1994, so, not too shabby.

Matt Raible: Developing with AngularJS - Part I: The Basics

2013-06-18T16:59:27+00:00

There's many, many different introductions to AngularJS available on the internet. This article is not another introduction, but rather a story about my learning experience. It all started way back in January of this year. I was working as a UI Architecture Consultant at Taleo/Oracle, my client for the last 21 months. My gig there ended last month, but they agreed to let me publish a series of articles about the knowledge I gained.

Project Background

The Director of Product Management had been working on the concepts for a new project - codenamed "Visual MyView". Below is a mockup he created for our kickoff meeting on January 4th.

From his original email about the above mockup:

The intent here is that one of the columns has rows that have a similar width. The rows could be dragged and dropped into a different order – or potentially the two columns could also be reordered. The rows will basically be comprised of similar widgets. You can see in the mockup how the first two rows might look – and sample widgets. The widgets shown can be configured by the end user, as well as the order in which they are displayed. Other requirements given to us were the following.

Row 1 is comprised of 'summary' widgets that are 'todo' items. Reviews needing done – approvals required – etc.
Row 2 will be a graph row – having graphs and charts to display information – larger squares will build this row.
Row 3's content was not determined yet.

I started the initial layout with static HTML and CSS and had a wireframe to show by mid January.

By the end of January, we'd renamed the project to My Dashboard and had a working prototype using CoolClock and moment.js for the clock in the top right, AngularJS to display widget data, jQuery UI for drag-n-drop of rows and widgets, Bootstrap's Carousel for holding charts and Highcharts for rendering charts. For this prototype, we included 4 types of widgets:

Summary
Tasks
Charts
Reports

To create widgets, we had to decide on a common schema for them.

"id": 1, // not necessary for display, but likely needed if we modify and save preferences
"title": "Appointments Today",
"type": "summary", // others include: task, chart, report
"value": 3, 
"description": "10:30 Jim Smith",
"events": "url", // this can have click events
"order": 1 // used to determine order

Below is a screenshot of our wireframe with some sample widgets.

Angular Basics

The decision to use AngularJS came early on in the project, after I read Tyler Renelle's Rant: Backbone, Angular, Meteor, Derby. To learn AngularJS, I briefly looked at its homepage documentation and played with some examples. Then I stumbled upon Misko Hevery and Igor Minar's AngularJS Presentation from Devoxx 2012. At that time, the video wasn't publicly available (it's free now), so I had to buy a Parley's subscription ($79). It was well worth the money because that one hour video greatly contributed to my understanding of how AngularJS works. Another resource I used frequently to figure out how to do things was John Lindquist's egghead.io.

To begin with, we wrote the JSON for a bunch of sample widgets and embedded them into the page as a widgetData JavaScript variable.

var widgetData = [
    {"id": 1, "title": "Appointments Today", "type": "summary", "value": 3, "description": "10:30 Jim Smith", "events": {"click": "alert('foo');"}, "order": 1},
    {"id": 12, "order": 2, "title": "Offer Approvals", "type": "task", "class": "sticky-note", "value": 1},
    {"id": 103, "title": "Browser market shares at a specific website, 2010", "order": 1, "type": "chart", "chartType": "pie", 
         "tooltip": {"pointFormat": "{series.name}: <b>{point.percentage}%</b>", "percentageDecimals": 1}, "series": [
         {"type": "pie", "name": "Browser share", "data": [
             ["Firefox", 45.0],
             ["IE", 26.8],
             {"name": "Chrome", "y": 12.8, "sliced": true, "selected": true},
             ["Safari", 8.5],
             ["Opera", 6.2],
             ["Others", 0.7]
         ]}
    ]},
    ...
];

I used angular-seed to create the initial structure of the prototype, and continued using the same JavaScript file names when we moved it into the product I worked on. Since the application takes a while to login and render the My Dashboard page (when working remotely), I decided not to use the Karma testing framework that ships with Angular. Below is what our directory structure looked like for our prototype.

The JavaScript files in the "js" folder are the most important for Angular. The first file, app.js, loads the other files:

angular.module('dashboard', ['dashboard.filters', 'dashboard.services', 'dashboard.directives']);

The controllers.js file contains the Controllers (functions) that get the data and make it available to the page. Here's the code for our first controller:

'use strict';

/* Controllers */
function WidgetController($scope) {
    $scope.widgets = widgetData;
}

This puts the widgets in scope and then we were able to render them using Angular's ngRepeat directive and the following HTML:

<div ng-app="dashboard" class="dashboard">
    <div class="container-widgets" ng-controller="WidgetController" ng-cloak>
        <div class="row-fluid">
            <div class="span9">
                <ul class="widgets">
                    <li id="summary-bar">
                        <div class="heading">Summary</div>
                        <ul class="tiles">
                            <li class="span3" ng-repeat="widget in widgets | filter:{type: 'summary'} | orderBy: 'order'">
                                <h3 class="events">{{widget.value}}</h3>
                                <div class="title">{{widget.title}}</div>
                                <div class="desc">{{widget.description}}</div>
                            </li>
                        </ul>
                    </li>
                    <li id="task-bar">
                        <div class="heading">My Tasks</div>
                        <ul class="tasks">
                            <li class="task {{widget.class}}" ng-repeat="widget in widgets | filter: {type: 'task'} | orderBy: 'order'">
                                <div class="title events">{{widget.title}}</div>
                                <div class="value">{{widget.value}}</div>
                            </li>
                        </ul>
                    </li>
                    <li id="chart-bar">
                        <div class="heading">Charts</div>
                        <div id="chartCarousel" class="carousel slide">
                            <ol class="carousel-indicators">
                                <li data-target="#chartCarousel"
                                    ng-repeat="widget in widgets | filter: {type: 'chart'} | orderBy: 'order'"
                                    data-slide-to="{{$index}}" ng-class="{active: $index == 0}"></li>
                            </ol>
                            <div class="carousel-inner">
                                <div class="item chart"
                                     ng-repeat="widget in widgets | filter: {type: 'chart'} | orderBy: 'order'"
                                     ng-class="{active: $index == 0}">
                                    <chart class="widget" value="{{widget}}" type="{{widget.chartType}}"></chart>
                                </div>
                            </div>
                            <a class="left carousel-control" href="#chartCarousel" data-slide="prev">‹</a>
                            <a class="right carousel-control" href="#chartCarousel" data-slide="next">›</a>
                        </div>
                    </li>
                </ul>
            </div>
            <div class="span3">
                <!-- clock and reports -->
            </div>
    </div>
</div>

The beginning of this HTML shows how Angular is instantiated: ng-app matches the name defined in app.js, ng-controller instantiates the WidgetController and ng-cloak is used to hide everything until its processed.

<div ng-app="dashboard" class="dashboard">
    <div class="container-widgets" ng-controller="WidgetController" ng-cloak>

If you take a closer look at the way ng-repeat attributes, you'll see how filters are used to filter data. There's filter and orderBy filters that are built in and allow you to filter data. The filter filter allows you to query arrays by strings, objects and even functions. In the following code block, "task" widgets are filtered, ordered and displayed.

<li class="task {{widget.class}}" ng-repeat="widget in widgets | filter: {type: 'task'} | orderBy: 'order'">
    <div class="title events">{{widget.title}}</div>
    <div class="value">{{widget.value}}</div>
</li>

This was pretty straightforward, but we quickly noticed that if a widget had HTML in its title, it didn't display correctly (rendering the raw HTML). To process the HTML, we had to use the ngBindHtml directive (tip: directives are camelCase, but written with dashes in HTML).

<div class="title events" ng-bind-html="widget.title"></div>

After getting this to work, we noticed that some titles weren't fully rendered because they were hidden with overflow: hidden. We tried adding a tooltip with title="{{widget.title}}", but ran into the same issue. I sent an email to the AngularJS Google Group and received a solution: create an htmlTitle directive:

.directive('htmlTitle', function ($sanitize) {
    return {
        restrict: 'A',
        link: function (scope, element, attrs) {
            attrs.$observe('htmlTitle', function (title) {
                // convert &value; to HTML
                var html = angular.element('<div></div>').html($sanitize(title)).text();
                element.attr('title', html);
                element.html(html);
            });
        }
    }
})

Usage:

<div class="title events" ng-bind-html="widget.title" html-title="{{widget.title}}"></div>

Drag-and-Drop

To implement drag-and-drop functionality, I originally used jQuery UI's sortable. At the bottom of the page, the following code initialized sorting for the various lists:

$(document).ready(function() {
    $('.widgets').sortable({
        cursor: "move",
        handle: ".heading"
    }).disableSelection();
    $('.tiles,.tasks').sortable();
    var carousel = $('.carousel');
    $(carousel).carousel({
        interval: 0
    });
};

As you can see, it also initializes the carousel and stops it from cycling automatically.

Carousel Issues

The first problem I ran into with Bootstrap's Carousel was a strange error from Highcharts. If you look in the above HTML, you'll see there's a <chart> element. This is processed by a highcharts directive. When I tried to use this directive for Highcharts in a carousel, it results in the following error:

TypeError: Cannot read property 'length' of undefined at Object.ob.setMaxTicks

This seemed to be caused by the following css in Bootstrap:

.carousel-inner > .item { display: none }

When I added an override with "display: block" to my stylesheet, everything worked, but the charts were stacked instead of in a carousel. To fix this, I modified the directive to show/hide the "item" element so Highcharts was able to write to it. I also logged an issue for this.

if (element.parent().not(':visible')) {
    element.parent().show();
}
var chart = new Highcharts.Chart(newSettings);
element.parent().attr('style', '');

ngRepeat and Grouping

The last thing I accomplished in our end-of-January prototype was rendering 2 charts side-by-side. I got it working with plain HTML, created a "groupBy" filter for Angular and tried to get it to work with the following:

<div id="chartCarousel" class="carousel slide">
    <ol class="carousel-indicators">
        <li data-target="#chartCarousel" ng-repeat="widget in widgets | filter: {type: 'chart'} | groupBy"
            data-slide-to="{{$index}}" ng-class="{active: $index == 0}"></li>
    </ol>
    <div class="carousel-inner">
        <div class="item" ng-repeat="widget in widgets | filter: {type: 'chart'} | groupBy" ng-class="{active: $index == 0}">
            <div class="widget">{{widget[0].title}}</div>
            <div class="widget">{{widget[1].title}}</div>
        </div>
    </div>
    <a class="left carousel-control" href="#chartCarousel" data-slide="prev">‹</a>
    <a class="right carousel-control" href="#chartCarousel" data-slide="next">›</a>
</div>

This all worked like I expected it to in Chrome, but I the following errors showed in my console.

Error: 10 $digest() iterations reached. Aborting! Watchers fired in the last 5 iterations:

I sent an email to the Angular Google Group and received a link to a discussion where I found a "chunk" filter that solved the problem. This worked great, but I wanted to make it more responsive.

If the user has a screen size big enough to fit 2 charts, show 2 charts and paginate by 2.
If the user has a small screen size that only fits 1 chart, show 1 and paginate by 1.

To solve #1 and #2, I ended up rendering two different sections (with classes .oneup and .twoup) and show/hide them based on screen size.

<li id="chart-bar">
    <div class="heading">Charts</div>
    <div id="chartCarousel1" class="carousel slide oneup" style="display: none">
        <ol class="carousel-indicators">
            <li data-target="#chartCarousel1"
                ng-repeat="widget in widgets | filter: {type: 'chart'} | orderBy: 'order'"
                data-slide-to="{{$index}}" ng-class="{active: $index == 0}"></li>
        </ol>
        <div class="carousel-inner">
            <div class="item chart"
                 ng-repeat="widget in widgets | filter: {type: 'chart'} | orderBy: 'order'"
                 ng-class="{active: $index == 0}">
                <chart class="widget" value="{{widget}}" type="{{widget.chartType}}"></chart>
            </div>
        </div>
        <a class="left carousel-control" href="#chartCarousel1" data-slide="prev">‹</a>
        <a class="right carousel-control" href="#chartCarousel1" data-slide="next">›</a>
    </div>
    <div id="chartCarousel2" class="carousel slide twoup" style="display: none">
        <ol class="carousel-indicators">
            <li data-target="#chartCarousel2"
                ng-repeat="widget in widgets | filter: {type: 'chart'} | chunk: 2 | orderBy: 'order'"
                data-slide-to="{{$index}}" ng-class="{active: $index == 0}"></li>
        </ol>
        <div class="carousel-inner">
            <div class="item chart"
                 ng-repeat="widget in widgets | filter: {type: 'chart'} | chunk: 2 | orderBy: 'order'"
                 ng-class="{active: $index == 0}">
                <chart class="widget" value="{{widget[0]}}" type="{{widget[0].chartType}}"></chart>
                <chart class="widget" value="{{widget[1]}}" type="{{widget[1].chartType}}"></chart>
            </div>
        </div>
        <a class="left carousel-control" href="#chartCarousel2" data-slide="prev">‹</a>
        <a class="right carousel-control" href="#chartCarousel2" data-slide="next">›</a>
    </div>
</li>

The JavaScript to show the correct number of charts is below:

var chartBar = $('#chart-bar');
function showCharts() {
    if (chartBar.width() < 960) {
        chartBar.find('.oneup').show();
        chartBar.find('.twoup').hide();
    } else {
        chartBar.find('.twoup').show();
        chartBar.find('.oneup').hide();
    }
}

$(document).ready(function () {
    showCharts();
});

$(window).resize(showCharts);

Summary

Even though I got everything to work for our initial prototype using Angular and jQuery, it didn't quite feel like I was taking full advantage of Angular's power. In particular, I learned that Angular UI Bootstrap had their own carousel and Angular UI had a sortable directive. My suspicions were confirmed when I read How do I "think in AngularJS" if I have a jQuery background?

In the next article, I'll talk about how I migrated to use Angular UI's carousel and sortable directives, as well as integrating dialogs.

Edward J. Yoon: 한국 IT 대기업들에 대한 아쉬움

2013-06-18T09:48:32+00:00

제프 마크험에게 들어서 이미 알고 있었지만 호튼웍스가 한국에 지사를 설립한다는 기사[1]를 오늘 보게 되었다.

내용 중 유독 눈에 띄는 문구가 있으니, ..

몇몇 국내회사가 하둡 분야의 오랜 경험을 보유했지만, 미국기업을 선호하는 국내의 기업용 솔루션 시장 풍토로 대우를 받지 못하는 상황이다.

내가 NHN이나 KT같은 대기업을 7~8년 간 재직하면서 대기업 소속 직장인들에게 느낀게 있다라면, 도전의식의 결여, 조잡한 경쟁의식, 위계질서, 책임전가, 집단이기주의 등 이다.

가령 내가, 우리네 대기업 개발자들이 필요한것을 개발하고 오픈소스화해서 한국 IT발전도 이끌자! 이러면, "회사에 무슨 이익을 주는가?", 또는 "이미 외국에 페이스북이나 트위터가 개발하고 그 기업에서 검증된 오픈소스를 쓰지 뭐하러?" 라는 질문들이 나오며, 최종 결정은 보통 제 3자에 의해 개발된 외국산 오픈 소스를 사용하는 방향으로 협의 된다.

어찌보면 생각해봐야할 문제처럼 보이긴 하는데, 그간 이런 논리로만 접근해서 한국 소프트웨어 산업이 이모냥 아니겠어. 비단 이런 예말고, 솔루션 도입이나 신사업 접근 방식도 이것과 모두 다르지 않다.

여기서 협의라는 표현을 쓴 이유는, 뭔가 under the desk 에서는 보이지 않는 분쟁이 있다는 얘기다. 언제나 그러한 분쟁 내면에는 도전의식의 결여, 조잡한 경쟁의식, 집단이기주의에서 오는 다양한 의도가 숨겨져 있지 (굳이 자세히 설명은 안함).

한편, "기업 이윤 추구"라는 큰 목적은 그 목적보다는 바텀업 문화와 창의성을 저해하는데 활용 된다. 무엇이 합리적이고 생산적인지 생각하는 것을 포기한 자들이 (물론 자신의 이익은 계산하겠지만) 주로 "그래서 회사에 무슨 정량적 이익을 주는가?" 란 공격으로 끝맺음 하려고 하지.

고등학교 때 교련 수업받은 세대가 슬슬 물갈이 되지 않는 한, 내 생각에 한국 IT 대기업들의 미래는 어둡기만 함. 쉬운 문제가 아니지만 "잡고있는 줄을 놔야 새로운 줄을 잡는다"는 것을 깨닫고, 우리네 개발자들 스스로 창조적 직업이라 믿는다면, 앞으로 우리의 의식 수준을 높여야 할거다. 똑같은넘 되지말고, 화이팅~

1. http://www.zdnet.co.kr/news/news_view.asp?artice_id=20130529081754

Stefane Fermigier: Présentation de la roadmap du GT Logiciel Libre

2013-06-18T08:27:45+00:00

Retrouvez la présentation de la roadmap du GT Logiciel Libre que j'ai donnée hier (17 juin 2013) à la convention du Pôle de compétitivité Systematic :

Roadmap du GT Logiciel Libre 2013-2020 from Stefane Fermigier

Stefane Fermigier: Discours de Fleur Pellerin sur le logiciel libre

2013-06-18T07:09:56+00:00

Le 13 juin dernier, Fleur Pellerin était invitée à l'inauguration des nouveaux bureaux de Mozilla à Paris, et y a prononcé le discours, très remarqué, suivant :

Que vous pouvez également télécharger directement (fichier .m4v, taille: 84Mo).

En voici la transcription :

Madame la présidente, chère Mitchell, madame la vice-présidente, chère Debbie,

Mesdames et messieurs, il y a vingt ans un grand chercheur a, je le cite, « pris le principe d’hypertexte et l’a relié au principe du TCP et du DNS, et alors boom, ce fut le World Wide Web. »

Et en 1993, donc, le CERN, cet organisme de recherche européen qui avait inventé le Web a décidé de donner cette invention au monde, en la distribuant sous ce qu’on n’appelait pas encore une « licence libre ». Ce choix, anodin en apparence, a changé la face du monde.

Il y a 10 ans, la fondation Mozilla naissait et quelques mois plus tard, cher Tristan Nitot, vous fondiez sa branche européenne. J’imagine combien cela doit vous faire plaisir d’être ici 10 ans après, dans ce magnifique hôtel particulier du XVIIIe siècle, ancienne demeure de l’ambassadeur d’Autriche, m’avez-vous dit tout à l’heure. Mozilla Firefox, construit sur l’ancien Netscape, et œuvrant inlassablement pour promouvoir les standards du Web, a aussi changé le Web et par là la manière dont nous nous informons et dont nous innovons.

Je suis donc très fière d’inaugurer ce soir les nouveaux bureaux de la fondation Mozilla à Paris. Pourquoi être venue inaugurer ces nouveaux locaux ?

D’abord en raison des valeurs de la fondation Mozilla, et du logiciel libre. Ces valeurs, ce sont l’accès à la connaissance pour tous, la confiance ou encore l’amplification des aspects d’intérêt publics d’internet. Ce sont aussi les valeurs sociales qui portent un modèle de société vertueux, ouvert, participatif, où toute donnée est d’abord considérée comme un bien accessible au plus grand nombre, et une source de connaissances que chacun peut utiliser, améliorer, partager. Le logiciel libre, les formats ouverts, c’est enfin une communauté de personnes qui constitue un véritable patrimoine de connaissances qu’est le code, sans cesse inachevé, toujours à enrichir. Au-delà des innovations et des technologies permises par le Web, des acteurs comme Wikimedia ou la fondation Mozilla ont démontré que l’innovation et le progrès peuvent aussi passer par le partage, l’absence de propriété. C’est une victoire essentielle sur les esprits qui nous permet aujourd’hui d’avancer dans d’autres domaines : je pense à l’open innovation ou à l’open data. Je sais que Mozilla n’est pas une formation politique, mais toutes ces valeurs résonnent particulièrement doux à une ministre de gauche comme moi, et je pense que le monde politique a des choses à apprendre de cette réussite.

Par ailleurs le logiciel libre est aussi un atout décisif pour notre économie. À plus d’un titre il permet d’abord de lutter contre les phénomènes de dépendance technologique envers tous ces acteurs qui sont propriétaires de nos outils informatiques quotidiens, et est donc un véritable garant de la souveraineté numérique. De plus comme on le voit aujourd’hui, et contrairement à certaines idées reçues, le libre et l’open source sont créateurs d’emploi. Des modèles d’affaire originaux ont été créés et c’est un facteur important de productivité et de compétitivité pour les entreprises et les administrations. En effet elles peuvent ainsi mieux maîtriser leurs patrimoines respectifs et concentrer leurs efforts sur ce qui représente pour elles la valeur ajoutée. Enfin le logiciel libre remet en cause les rentes de situation, peu favorables à l’innovation, et par là-même aide à l’émergence de nouveaux champions économiques. L’émergence de Firefox et des navigateurs est emblématique de cette capacité.

La France est souvent citée comme un des pays les plus actifs au monde dans le domaine du logiciel libre. La croissance soutenue dans ce secteur le confirme. Les chiffres sont éloquents : ce marché représentait en 2011 plus de 2 milliards d’euros, soit plus de 6% de la demande de logiciels et de services informatiques. Par ailleurs, il y a là un formidable levier d’emplois, environ 10 000 supplémentaires dans les 3 ans à venir, si les estimations de croissance du marché sont confirmées. Bref, ce sont des enjeux extrêmement importants. La décision de Mozilla, un acteur mondial de référence sur le logiciel libre, de s’implanter dans ces locaux, confirme l’attractivité de Paris comme place incontournable du numérique. Elle confirme l’excellence des formations françaises dans le domaine informatique et je suis certaine que les développeurs du monde entier seront attirés par les conditions d’accueil ici, dans ces magnifiques locaux.

Mon objectif est bien sûr de renforcer encore ces atouts avec notamment le projet de quartiers numériques que nous allons créer dans une quinzaine de villes. Enfin, nous avons en France une communauté parmi les plus dynamiques dans le monde pour la conception et l’utilisation des logiciels libres, un atout à évidemment ne pas négliger. Avec les Assises de l’Entreprenariat, nous avons souhaité aller encore plus loin pour renforcer notre attractivité en mobilisant notamment toutes les compétences de France et d’ailleurs. C’est pour accompagner ce mouvement que le gouvernement travaille ardemment à la création d’un visa entrepreneur et d’un visa talent, car il est impératif d’attirer les talents créateurs du monde entier en leur offrant des conditions d’installation très rapides et simplifiées. Je dois aussi rappeler que le gouvernement prête une attention toute particulière à l’utilisation des logiciels libres. Par notre action nous visons à la renforcer. Le recours au logiciel libre est un levier d’action pour moderniser et rationaliser l’action publique.

Le Libre n’est pas toujours la bonne ou la seule solution mais la circulaire du Premier Ministre de septembre 2012 concernant l’utilisation des logiciels libres dans l’administration fixe une ligne claire sur les cas où ces types de logiciels doivent être privilégiés. Les atouts du logiciel libre sont notamment, je cite, « un moindre coût, une souplesse d’utilisation, et un levier de discussion avec les éditeurs ». Il s’agit là d’une avancée majeure pour le logiciel libre dans les systèmes d’information de l’État, qui permet d’engager de véritables politiques publiques en matière de logiciel libre et d’open source.

Pour conclure, je voudrais juste insister sur un point important que j’ai déjà évoqué mais que je tiens à marteler : l’open source est avant tout un vecteur d’innovation et de changement, un véritable gisement de productivité et de compétitivité pour les entreprises, et garantit la pérennité et l’indépendance de l’État. C’est pour cela que je souhaite que la France continue de jouer un rôle moteur dans le développement de ce secteur, et je suis sûre que la présence de Mozilla à Paris nous y aidera. J’ai parlé de 1993, de 2003, quoi de neuf en 2013 ? Cette année Mozilla lance son système d’exploitation pour mobiles et tablettes, Firefox OS, sur le même constat : promouvoir les formats ouverts et empêcher les systèmes fermés de contrôler notre environnement informatique. C’est une ambition un peu folle, mais probablement pas plus folle que de s’attaquer au marché des navigateurs qui était contrôlé à 95% par un seul acteur. Je vous souhaite donc le même succès que Firefox, et comme il sera en partie développé ici, je n’ai pas de doute sur la réussite de ce projet. Merci à tous.

Crédits: vidéo: Roberto Di Cosmo, montage de la vidéo: Stefane Fermigier, transcription du discours: Goofy, aKa, Z, Asta, Peekmo pour Framablog.

Matt Raible: Father's Day Weekend at The Oregon Coast

2013-06-18T06:00:10+00:00

For the 5-year anniversary of our Father's Day Camping Trip, we decided to mix things up a bit. My parents were in Oregon for a friend's wedding, so we decided to fly to meet them there instead of flying them to Colorado. My sister hopped on "Buttercup" (her Harley) and drove to meet us from Washington. We also invited our good friends Clint and Autumn (who you might remember from Costa Rica) and had a great time staying at Gearhart Cottages.

We didn't feel too bad about skipping camping since Trish held a "Boot Camp" with the kids and their friends at Stillwater Campground the week before. You might remember Stillwater from last year's Father's Day trip.

Trish's Uncle John drove out to the coast to meet us and we did a really fun hike along Indian Beach to Ecola State Park. It was a real treat! It was also really fun to meet Clint and Autumn's 9 month old Brodie. As usual, Trish snapped some incredible photos.

That was two weeks ago. We returned to Colorado, had some fun at ÜberConf, rocked out at a Def Leppard concert then enjoyed a few Rockies games before hopping in the car and heading to Montana. We spent the night at Roosevelt Lodge Cabins in Yellowstone and saw a plethora of buffalo, a couple bears and even a beaver while driving through.

This morning, we woke up at the cabin I was born in.

We'll be working on my parent's retirement cabin, getting a float ready for the Swan Valley Parade and having a good ol' time with great friends for the next two weeks. After that, we're taking the kids to Hawaii for the first time.

Here's to summer vacations. I hope yours is spent with great people in beautiful places too.

Justin Mason: Links for 2013-06-17

2013-06-17T23:58:02+00:00

Atelier olschinsky – “Cities III 05″

Fine Art Print on Hahnemuehle Photo Rag Bright White, 310g: 40x50cm up to 70x100cm. Some great art based on decayed urban landscape shots, from a Vienna-based design studio. See also http://english.mashkulture.net/2011/10/17/atelier-olschinsky-cities-iii/ , http://www.mascontext.com/tag/atelier-olschinsky/

(tags: olschinsky cities urban decay landscape art prints want)
Possible ban on ‘factory food’ in French restaurants

I am very much in favour of this in Ireland, too. The pre-prepared food thing makes for crappy food:
In an attempt to crack down on the proliferation of restaurants serving boil-in-a-bag or microwave-ready meals, which could harm France’s reputation for good food, MP Daniel Fasquelle is putting a new law to parliament this month. [...] The proposed law would limit the right to use the word “restaurant” to eateries where food is prepared on site using raw ingredients, either fresh or frozen. Exceptions would be made for some prepared products, such as bread, charcuterie and ice cream.

(tags: restaurants food france cuisine boil-in-the-bag microwave cooking daniel-fasquelle)
On Scala

great, comprehensive review of the language, its pros and misfeatures, from Bill de hOra

(tags: scala languages coding fp reviews)
Introducing Kale « Code as Craft

Etsy have implemented a tool to perform auto-correlation of service metrics, and detection of deviation from historic norms:
at Etsy, we really love to make graphs. We graph everything! Anywhere we can slap a StatsD call, we do. As a result, we’ve found ourselves with over a quarter million distinct metrics. That’s far too many graphs for a team of 150 engineers to watch all day long! And even if you group metrics into dashboards, that’s still an awful lot of dashboards if you want complete coverage. Of course, if a graph isn’t being watched, it might misbehave and no one would know about it. And even if someone caught it, lots of other graphs might be misbehaving in similar ways, and chances are low that folks would make the connection. We’d like to introduce you to the Kale stack, which is our attempt to fix both of these problems. It consists of two parts: Skyline and Oculus. We first use Skyline to detect anomalous metrics. Then, we search for that metric in Oculus, to see if any other metrics look similar. At that point, we can make an informed diagnosis and hopefully fix the problem.
It’ll be interesting to see if they can get this working well. I’ve found it can be tricky to get working with low false positives, without massive volume to “smooth out” spikes caused by normal activity. Amazon had one particularly successful version driving severity-1 order drop alarms, but it used massive event volumes and still had periodic false positives. Skyline looks like it will alarm on a single anomalous data point, and in the comments Abe notes “our algorithms err on the side of noise and so alerting would be very noisy.”

(tags: etsy monitoring service-metrics alarming deviation correlation data search graphs oculus skyline kale false-positives)
Paper: “Root Cause Detection in a Service-Oriented Architecture” [pdf]

LinkedIn have implemented an automated root-cause detection system:
This paper introduces MonitorRank, an algorithm that can reduce the time, domain knowledge, and human effort required to ?nd the root causes of anomalies in such service-oriented architectures. In the event of an anomaly, MonitorRank provides a ranked order list of possible root causes for monitoring teams to investigate. MonitorRank uses the historical and current time-series metrics of each sensor as its input, along with the call graph generated between sensors to build an unsupervised model for ranking. Experiments on real production outage data from LinkedIn, one of the largest online social networks, shows a 26% to 51% improvement in mean average precision in ?nding root causes compared to baseline and current state-of-the-art methods.
This is a topic close to my heart after working on something similar for 3 years in Amazon! Looks interesting, although (a) I would have liked to see more case studies and examples of “real world” outages it helped with; and (b) it’s very much a machine-learning paper rather than a systems one, and there is no discussion of fault tolerance in the design of the detection system, which would leave me worried that in the case of a large-scale outage event, the system itself will disappear when its help is most vital. (This was a major design influence on our team’s work.) Overall, particularly given those 2 issues, I suspect it’s not in production yet. Ours certainly was ;)

(tags: linkedin soa root-cause alarming correlation service-metrics machine-learning graphs monitoring)
Announcing Zuul: Edge Service in the Cloud

Netflix’ library to implement “edge services” — ie. a front end to their API, web servers, and streaming servers. Some interesting features: dynamic filtering using Groovy scripts; Hystrix for software load balancing, fault tolerance, and error handling for originated HTTP requests; fine-grained service metrics; Archaius for configuration; and canary requests to detect overload risks. Pretty complex though

(tags: edge-services api netflix zuul archaius canary-requests http groovy hystrix load-balancing fault-tolerance error-handling configuration)
CloudFlare, PRISM, and Securing SSL Ciphers

Matthew Prince of CloudFlare has an interesting theory on the NSA’s capabilities:
It is not inconceivable that the NSA has data centers full of specialized hardware optimized for SSL key breaking. According to data shared with us from a survey of SSL keys used by various websites, the majority of web companies were using 1024-bit SSL ciphers and RSA-based encryption through 2012. Given enough specialized hardware, it is within the realm of possibility that the NSA could within a reasonable period of time reverse engineer 1024-bit SSL keys for certain web companies. If they’d been recording the traffic to these web companies, they could then use the broken key to go back and decrypt all the transactions. While this seems like a compelling theory, ultimately, we remain skeptical this is how the PRISM program described in the slides actually works. Cracking 1024-bit keys would be a big deal and likely involve some cutting-edge cryptography and computational power, even for the NSA. The largest SSL key that is known to have been broken to date is 768 bits long. While that was 4 years ago, and the NSA undoubtedly has some of the best cryptographers in the world, it’s still a considerable distance from 768 bits to 1024 bits — especially given the slide suggests Microsoft’s key would have to had been broken back in 2007. Moreover, the slide showing the dates on which “collection began” for various companies also puts the cost of the program at $20M/year. That may sound like a lot of money, but it is not for an undertaking like this. Just the power necessary to run the server farm needed to break a 1024-bit key would likely cost in excess of $20M/year. While the NSA may have broken 1024-bit SSL keys as part of some other program, if the slide is accurate and complete, we think it’s highly unlikely they did so as part of the PRISM program. A not particularly glamorous alternative theory is that the NSA didn’t break the SSL key but instead just cajoled rogue employees at firms with access to the private keys — whether the companies themselves, partners they’d shared the keys with, or the certificate authorities who issued the keys in the first place — to turn them over. That very well may be possible on a budget of $20M/year. [....] Google is a notable anomaly. The company uses a 1024-bit key, but, unlike all the other companies listed above, rather than using a default cipher suite based on the RSA encryption algorithm, they instead prefer the Elliptic Curve Diffie-Hellman Ephemeral (ECDHE) cipher suites. Without going into the technical details, a key difference of ECDHE is that they use a different private key for each user’s session. This means that if the NSA, or anyone else, is recording encrypted traffic, they cannot break one private key and read all historical transactions with Google. The NSA would have to break the private key generated for each session, which, in Google’s case, is unique to each user and regenerated for each user at least every 28-hours. While ECDHE arguably already puts Google at the head of the pack for web transaction security, to further augment security Google has publicly announced that they will be increasing their key length to 2048-bit by the end of 2013. Assuming the company continues to prefer the ECDHE cipher suites, this will put Google at the cutting edge of web transaction security.
2048-bit ECDHE sounds like the way to go, and CloudFlare now support that too.

(tags: prism security nsa cloudflare ssl tls ecdhe elliptic-curve crypto rsa key-lengths)
Record companies to target 20 more pirate sites after court ruling – Independent.ie

Looks like IRMA are following the lead of the UK’s BPI, by chasing the proxy sites next:
Up to 20 internet sites are to be targeted by an organisation representing record companies in a move to stamp out the illegal pirating of music and other copyright material. The Irish Recorded Music Association (IRMA) said it would be immediately moving against the 20 “worst offenders” to “take out” internet sites involved in the illegal downloading of copyright work.
However, looks like this will involve more court time:
Last night IRMA director general, Dick Doyle said the High Court ruling was only the first step in “taking out many internet sites involved in illegally downloading music. “We will be back in court very shortly to take out five to 10 other sites. We have already selected a total of 20 of the worst offender sites and we will go after the next five in the very near future,” he said.
That’s not going to be cheap!

(tags: courts ireland law irma piracy pirate-bay bpi proxies filesharing copyright)
Building a Modern Website for Scale (QCon NY 2013) [slides]

some great scalability ideas from LinkedIn. Particularly interesting are the best practices suggested for scaling web services: 1. store client-call timeouts and SLAs in Zookeeper for each REST endpoint; 2. isolate backend calls using async/threadpools; 3. cancel work on failures; 4. avoid sending requests to GC’ing hosts; 5. rate limits on the server. #4 is particularly cool. They do this using a “GC scout” request before every “real” request; a cheap TCP request to a dedicated “scout” Netty port, which replies near-instantly. If it comes back with a 1-packet response within 1 millisecond, send the real request, else fail over immediately to the next host in the failover set. There’s still a potential race condition where the “GC scout” can be achieved quickly, then a GC starts just before the “real” request is issued. But the incidence of GC-blocking-request is probably massively reduced. It also helps against packet loss on the rack or server host, since packet loss will cause the drop of one of the TCP packets, and the TCP retransmit timeout will certainly be higher than 1ms, causing the deadline to be missed. (UDP would probably work just as well, for this reason.) However, in the case of packet loss in the client’s network vicinity, it will be vital to still attempt to send the request to the final host in the failover set regardless of a GC-scout failure, otherwise all requests may be skipped. The GC-scout system also helps balance request load off heavily-loaded hosts, or hosts with poor performance for other reasons; they’ll fail to achieve their 1 msec deadline and the request will be shunted off elsewhere. For service APIs with real low-latency requirements, this is a great idea.

(tags: gc-scout gc java scaling scalability linkedin qcon async threadpools rest slas timeouts networking distcomp netty tcp udp failover fault-tolerance packet-loss)
Why I won’t give the European Parliament the data protection analysis it wanted

Holy crap. Simon Davies rips into the EU data-protection reform disaster with gusto:
The situation was an utter disgrace. The advertising industry even gave an award to an Irish Minister for destroying some of the rights in the regulation while the UK managed to force a provision that would make the direct marketing industry a “legitimate” processing operation in its own right, putting it on the same level of lawful processing as fraud prevention. Things got to the point where even the most senior data protection officials in Europe stopped trying to influence events and had told me “let the chips fall as they may”. [...] But let’s take a step back for a moment from this travesty. Out on the streets – while most may not know what data protection is – people certainly know what it is supposed to protect. People value their privacy and they will be vocal about attempts to destroy it. I had said as much to the joint parliamentary meeting, observing “the one element that has been left out of all these efforts is the public”. However, as the months rolled on, the only message being sent to the public was that data protection is an anachronism stitched together with self interest and impracticality. [...] I wasn’t aware at the time that there was a vast stitch-up to kill the reforms. I cannot bring myself to present a temperate report with measured wording that pretends this is all just normal business. It isn’t normal business, and it should never be normal business in any civilized society. How does one talk in measured tones about such endemic hypocrisy and deception? If you want to know who the real enemy of privacy is, don’t just look to the American agencies. The real enemy is right here in the European Parliament in the guise of MEPs who have knowingly sold our rights away to maintain powerful relationships. I’d like to say they were merely hoodwinked into supporting the vandalism, but many are smart people who knew exactly what they were doing.
Nice work, Irish presidency! His bottom line:
Is there a way forward? I believe so. First, governments should yield to common decency and scrap the illegitimate and poisoned Irish Council draft and hand the task to the Lithuanian Presidency that commences next month. Second, the Irish and British governments should be infinitely more transparent about their cooperation with intrusive interests that fuelled the deception.

(tags: ireland eu europe reform law data-protection privacy simon-davies meps iab)
Persuading David Simon (Pinboard Blog)

Maciej Ceglowski with a strongly-argued rebuttal of David Simon’s post about the NSA’s PRISM. This point in particular is key:
The point is, you don’t need human investigators to find leads, you can have the algorithms do it [based on the call graph or network of who-calls-who]. They will find people of interest, assemble the watch lists, and flag whomever you like for further tracking. And since the number of actual terrorists is very, very, very small, the output of these algorithms will consist overwhelmingly of false positives.

(tags: false-positives maciej privacy security nsa prism david-simon accuracy big-data filtering anti-spam)
Schneier on Security: Blowback from the NSA Surveillance

Unintended consequences on US-focused governance of the internet and cloud computing:
Writing about the new Internet nationalism, I talked about the ITU meeting in Dubai last fall, and the attempt of some countries to wrest control of the Internet from the US. That movement just got a huge PR boost. Now, when countries like Russia and Iran say the US is simply too untrustworthy to manage the Internet, no one will be able to argue. We can’t fight for Internet freedom around the world, then turn around and destroy it back home. Even if we don’t see the contradiction, the rest of the world does.

(tags: internet freedom cloud-computing amazon google hosting usa us-politics prism nsa surveillance)

Joe Brockmeier: Get Ready to Hack at the Collaboration Conference

2013-06-17T20:54:00+00:00

This coming Sunday (June 23rd) we'll be getting together at the CloudStack Collaboration Conference for hack day, from 9:30 a.m. to 5:00 p.m. in the Santa Clara Convention Center.

The hack day is being run using the BarCamp/unconference format – meaning that the sessions will be chosen Sunday by the folks who show up. Sessions might run an hour or all day, depending on how much interest there is and how much needs to be done.

Have an idea for a session? A topic you want to lead or see discussed? Add it to the Hack Day at CCC13 page on the CloudStack Wiki. It is, of course, a wiki: so edit boldly!

Don't let the "hack day" moniker fool you – non-development topics and sessions are more than welcome. Giles Sirett has proposed a marketing session, and we'll likely have at least one documentation session as well. If it's CloudStack-related, it's fair game.

Pleased to see that we've got a pretty good selection of sessions already proposed, but we can always do with more. If you have an idea, put 'er up and show up Sunday ready to jam with the rest of the CloudStack community!

Rich Bowen: VuPoint Magic Wand portable scanner

2013-06-17T20:16:14+00:00

I just bought, returned, and bought VuPoint MagicWand scanners.

Here's the story.

The first one I bought was this one, which is a hand-held scanner wand thingy that scans to a micro SD card. That part works great. The other feature was that you could transfer the files wirelessly. There wasn't much information as to what that actually means, so it sounded like a good idea.

Turns out that what it means is that the device advertises its own SSID, which you must connect to, and then open a web site on the device which lists the files. I don't know if it was browser incompatibility or what, but the list of files didn't actually link to the files - just listed them. Viewing the source, it certainly had the links in the HTML, but something was preventing those links from actually being displayed on the page. Weird.

So, having to disconnect from my work VPN to connect to my scanner isn't exactly a great idea. Also, the network has a fixed name (MAGICWAND) and password (123456789), which ensures that as soon as I switch it on, everyone in range has access to my images. Not exactly idea.

So, I returned it and got the one without wifi. There's a USB cable, or the SD card can just go into a normal SD card reader. Only a $20 price difference, but I don't care to pay $20 for functionality I don't want.

Bryan Pendleton: Tom Fassbender's JMT hike

2013-06-17T16:13:36+00:00

I'm completely loving Tom Fassbender's awesome series of articles about his three week epic hike on the John Muir Trail in June, 2012.

His writing style is great; I love the links to gear, locations, and background information, and the pictures are a lot of fun, too!

Here are the articles he's published so far, with more on the way (I hope):

John Muir Trail Adventure Journal: Day One
I talked briefly to a group of hikers headed toward Yosemite. They were complaining of the mosquitoes, which I didn’t think were that bad. (But they were going to get a lot worse before this hike was over).
John Muir Trail Adventure Journal: Day Two
I thought for a moment that maybe it was a bear until it heard me and looked up. It was just a mule deer. The deer along the trail, especially the does, proved to be almost totally unafraid of humans. It was kind of like being in a petting zoo.
John Muir Trail Adventure Journal: Day Three
A bit about trail names.
There are rules to these things.
1. You can’t name yourself.
2. You can’t decline a trail name.
3. It should have some meaning and/or describe you, be it a good thing, a bad thing, or just a humorous thing.
4. Once you have one, you’re stuck with it for life and it follows you from trail to trail
Not all who have trail names abide by these rules, but everyone I talked to who used a trail name earned them by these rules.
John Muir Trail Adventure Journal: Day Four
As I crossed a bridge into the heart of the monument, I ran into a day hiker from the nearby campsite. He was the first person I’d seen in about 24 hours. It had been a strange kind of solitude that I didn’t even think about until I saw this guy.
John Muir Trail Adventure Journal: Day Five
After a relentless climb and countless downed trees, I made it to as series of lakes named for what white people thought of as traditional roles for the Native American people.
John Muir Trail Adventure Journal: Day Six
I talked with other hikers. We talked trail conditions, the gear we used, crazy things we’d seen, other hikers we’d met. This is when I learned about Tuna Helper‘s bid to set a PCT speed record. There was also talk of a hiker who wore different wedding dresses along the trail. He’d wear an old wedding dress for a certain amount of distance depending on how much you paid him. You’d get a picture of course. Somehow we missed each other on the trail. I would have enjoyed seeing that.
John Muir Trail Adventure Journal: Day Seven
I skipped my usual lunch, instead enjoying Fritos and Peanut M&Ms. This is the point when the meals started blending together, and I’d switch it up (like soup for breakfast and oatmeal for dinner) to keep things from getting too boring.
A bit about Fritos
One of the greatest hiking foods of all time! Per bag, 1,760 calories, 990 of that from fat. Plus a good amount of sodium. These things keep your motor running! And they’re great kindling.

Looking forward to the full report, Tom!

Bryan Pendleton: Shear Keys

2013-06-17T15:46:43+00:00

We're supposed to be in the final home stretch for opening the new Bay Bridge, and this weekend the New York Times carried this story: In California, Bolts May Hold Up a Bridge in More Ways Than One

The new bridge, whose foundation will reach the bedrock underneath, will be the world’s largest self-anchored suspension bridge, an unusual design that contributed to its high cost. Equipped with the latest antiseismic technology, the bridge was designed to last 150 years by remaining intact in a major earthquake.
That is why the failure of some high-strength steel bolts attaching shock-absorbing devices called shear keys to a concrete crossbeam under the roadway raised alarms. When workers tightened the 17- to 24-foot-long bolts in March, 32 in a batch of 96 snapped. Engineers blamed hydrogen-assisted cracking, in which atoms of hydrogen infiltrate steel and make it brittle, but they have yet to determine its cause.

Don't miss the nifty infographic: Solving a Structural Issue, which explains the concept of the Shear Keys with a nice diagram.

Shear keys and bearings allow limited and controlled movement at the joints of different sections of the bridge during seismic events.

Of course, since the whole point of the new bridge is to address seismic issues, it's natural that engineers are intently focused on the seismic aspects of the new design.

I've long been fascinated with this part of civil engineering, since I live in such a seismically active area. Ten years ago, I was privileged to have a window seat for the construction of Oakland's Cathedral of Christ the Light, a beautiful building, and once again a structure which was constructed to replace a structure that collapsed in the same great Loma Prieta earthquake.

Among the interesting aspects of the Cathedral of Christ the Light is the use of enormous wooden beams for the main space, since the natural flexibility of the wooden beams is supposed to make a more earthquake-friendly structure. Interesting, since as the NY Times article about the bridge observes,

the eastern half, a cantilever design that runs from the island to Oakland, was deemed beyond retrofitting. Its main problem was that it was anchored in treated Douglas fir trees that were used as pilings in clay and mud.

Poor trees: sometimes they are the blessing; sometimes they are the curse.

A more common earthquake technique, used in both the bridge and the cathedral, is the use of special seismic bearings. In the cathedral, as the manufacturer points out,

The new cathedral is located just 4.7 km from the Hayward fault and uses Friction PendulumTM seismic isolation bearings to protect it from damage during a 1000 year earthquake. The bearings support a 120 foot tall main sanctuary, which incorporates glulaminated timber, reinforced concrete, high strength steel, and aluminum and glass to create a light-filled structure.

I remember watching the bearings being installed; they were quite fasinating to observe.

And, of course, don't find the language confusing: "a 1000 year earthquake" is not an earthquake which lasts for 1000 years (though in fact the faults in California are nearly constantly moving, and it's isn't an inaccuracy to describe such continual movements in such a way), but rather an earthquake of such severity that we might expect it would occur once every 1000 years.

Meanwhile, back over the water, the engineers are pointing out that nothing is ever perfectly safe, and rather than ask whether the new bridge is flawless, we should be asking whether it is (a) safe enough, and (b) safer than the current bridge: Bay Bridge: Open new span with or without bolt repairs, experts say

If a moderate temblor were to strike after traffic moves onto the new bridge but before the retrofit is done or any other bolts are replaced, the new bridge is still at least twice as safe as the 77-year-old span motorists use today, Seible and Fisher said.

And some engineers just love the fact that the Bay Area is building new bridges: My Word: Stop all the negativity about beautiful bridge project

We in the Bay Area are renowned for our creativity, our leadership in technology and our eagerness to move forward. This bridge exemplifies all of these strengths. And let's not forget that Caltrans originally proposed a conservative, although arguably bland, simple viaduct structure to replace the existing bridge.
We rejected that and are much better off for it.
Although there are no guarantees in seismic engineering because temblors are inherently random, computer modeling and analysis has become so sophisticated that we have a truly remarkable understanding of the predicted performance of large structures in seismic events.

There's no doubt that the attention on seismic design has brought amazing improvements in construction techniques. But I'm also pleased to see that there's always room for improvement, and people are already actively studying the results of these recent large public projects, trying to learn from them and figure out how to do still better in the future.

Rich Bowen: Open Help Conference, 2013

2013-06-17T14:35:23+00:00

This weekend I attended the OpenHelp Conference in Cincinnati. Unfortunately, I was only able to go to one day of it, as we had to be back for something Sunday morning.

It was smallish, and so there was a lot of good conversation and brainstorming.

The focus was both documentation and support, which are, of course, deeply intertwingled. It gave me a lot to think about, and I really wish I could have been there for the second day as well.

Siobhan McKeown, from the Wordpress documentation team, was at the conference, and took amazing notes, so I'm going to link to her writeup for each talk.

The day started with Jorge Castro talking about using StackExchange to handle the Question & Answer part of support.StackExchange is part of the StackOverflow family of sites, Each StackExchange site is focused on a particular community, and is very focused on Question and Answer format, rather than general discussion. It allows users to vote for the quality of questions and answers, and seems to be a great way to get the subject matter experts more directly involved in the support process.

Siobhan's notes are here.

Following that, Michael Verdi, from the Firefox support team, talked about the SuMo site and the work that they had done to help users find the answers to their question. Of particular interest was some graphs he showed of the improvement in customer satisfaction, as well as the rate of answered questions, brought about by just improving the search functionality, to help users find the right docs so that they didn't even need to ask their question.

Firefox has their own home-grown, but Open Source, solution, called Kitsune. It has some StackExchage-like features, and also has a great tool called Army of Awesome, which is a way to watch Twitter mentions of your project/product, and ensure that at least one person from the expert community has responded to each one.

Here's Siobhan's notes.

This was followed by a panel discussion including Jorge, Michael, Jeremy Garcia (LinuxQuestions.org, and Siko Bouterse from Wikipedia. The discussion ranged from Wikipedia author retention to further discussion of many of the issues that Michael and Jorge had raised.

I spoke next, talking about listening to your audience. This is something I've thought a lot about over the years. My trepidation in speaking at this conference was that it seems like many of the people there know a lot more about documentation and support than I do, as I'm largely self-taught in this area. But it seemed that my remarks were well received. Once again here's Siobhan's notes, which in this case are way better than my own notes for my talk.

I was the last speaker of the day, and this was followed by a general discussion of the things that had been raised during the day, as well as many related issues.

You can see a lot of commentary about the events of the day, and of Sunday, by looking at the #openhelp keyword on Twitter. I'm looking forward to reading Siobhan's notes from Sunday's sessions.

Claus Ibsen: Introducing the new Apache Camel based iPaaS

2013-06-17T09:21:16+00:00

I just got back from CamelOne 2013 which was the 3rd annual Camel conference.

At the conference, James Strachan, introduced the new open source Apache Camel based iPaaS at the ending keynote on the first day.

You can find more details from James blog where he put out this information with the slides and a cool 20 minute video demonstrating the iPaaS in action. Its all a single page HTML5 web console, using the awesome hawio project.

In the video you will also see cool stuff such as the new Camel visual debugger, where you can set breakpoints, and single step the message in the routes etc.

And we also reveal for the first time the new web UI for log analytics (insight search). So you easily search in the logs from all the containers, and correlate the log messages, and have all that visualized with timeline, graphs etc.

I guess a few screenshot is worth a 1000 words in this case.

Lets start with the new insight plugin in action ...

Fabric Insight in action - Live log analytics using Kibana web UI, to aggregate and visualize log from all your containers. Notice the Camel tab with first class Camel integration.

And the debugger in action ...

Camel debugger in action - Live debugging of Camel routes, set breakpoint, single step, see message details. And as well edit the message on the fly (to be added)

And we also got time to add a new Camel route creator / editor that is purely web based ...

Camel route designer - A full Camel editor in the web console with wizards, making it easier for casual Camel developers to use.

Though the video shows all this in action, and there is a funky theme song to go along. The selection of the song was done after careful consideration in the bar, after a long day at CamelOne ;)

To see the video I suggest to click this link to get to James blog, or use this link for the direct link to Vimeo where the video is hosted.

On James blog you will also find instructions how to run this from the source code and try this on your own.

Fuse Fabric and hawtio are both 100% open source projects, with the ASL 2.0 license, and their source code hosted on github. We love contributions and collaboration. So feel free to fork and send pull requests. And if you send too many, then you can become a member of these projects, and commit directly ;)

Francesco Chicchiricco: Apache Syncope 1.1.2 (Ad libitum) released

2013-06-17T08:07:00+00:00

The Apache Syncope team is proud to announce the availability of the new maintenance release of Ad libitum.

Full story »

Original post blogged on b2evolution.

James Strachan: introducing the Apache Camel based open source iPaaS

2013-06-17T07:28:08+00:00

I've had a blast at CamelOne and Red Hat Summit in Boston; thanks for all the great speakers and feedback!

I demo'd our new Apache Camel based iPaaS (i.e. our Camel Cloud or Fuse integration appliance). Here's a quick demo video of what I presented at my CamelOne keynote along with the slides.

If you don't mind working on the bleeding edge here are the instructions for building and running it, though we'll have the public stable early access release soon.

CamelOne Keynote: introducing the Apache Camel based open source iPaaS from strachaj

Camel in the cloud demo from CamelOne 2013 from JBoss Developer on Vimeo.

Shalin Shekhar Mangar: First Bangalore Lucene/Solr Meetup Report

2013-06-17T06:01:00+00:00

The First Bangalore Lucene/Solr meetup was organized on Saturday, 8th June 2013 courtesy of the initiatives of Anshum Gupta and Varun Thacker. Although I joined in as a co-organizer but honestly I did nothing except tweet about it and show up with some slides.

I must say that I was pleasantly surprised at the rate at which the group went from zero to a hundred members (it stands at a 132 members as of writing this post). Our initial limit for the venue was fifty but it was increased to seventy five once the size of the venue was confirmed. Microsoft Accelerator was gracious enough to provide a conference room and refreshments for the attendees. 50+ people showed up which is pretty good considering that the meetup schedule clashed with some other popular meetups. The agenda was dominated by presentations but quite a bit of time was spent in Q&A.

Vinodh Kumar R (Head of BloomReach India) gave a talk on ranking models (adversarial vs implicit vs real time news) applicable for different kind of search applications.

Varun Thacker from Unbxd talked about Faceted search and result reordering in Solr focusing on e-commerce applications. He introduced term range facets, multi-select faceting and then delved into reordering documents using function queries and query elevation along with examples and use-cases for each.

Dikchant Sahi presented Apache Solr’s DataImportHandler to index databases and xml files in Solr. He also gave a live demo of full and delta imports of a sample music data set into Solr.

I gave a presentation on SolrCloud and Shard Splitting which is something that Anshum Gupta and I have been working on the past few months.

Here are the slides that I presented:

SolrCloud and Shard Splitting from Shalin Mangar

At the start of my presentation, I solicited an informal poll from the audience to gauge their interest:

Everyone in attendance was familiar with the projects
Most of the attendees were already using Solr for search
Everyone in attendance was using Solr 3.x and no one was on SolrCloud yet
Almost everyone was evaluating, prototyping or testing SolrCloud

I met Umesh Prasad from Flipkart at the meetup and we chatted quite a bit on Solr’s performance under heavy bulk re-indexing workloads and also about accommodating large elevation and synonym files in SolrCloud. I’m happy to know that Flipkart uses Apache Solr for their excellent search. I also met a couple of search enthusiasts who have used Solr in the past and want to contribute back to the community.

All in all, I think it was a good first step towards establishing a strong Lucene/Solr community in Bangalore. I wish that the next meetup gives more time for one to one interactions and focused conversations around search issues. It’d be nice to have more about Lucene in the next meetup. A lot of people inquired about training on Apache Solr so we may organize a workshop for the next meetup.

Drop me a line if you are interested in attending a Solr training in Bangalore. Also, if you’re in Bangalore and interested in Lucene/Solr or search in general, do join the Bangalore Lucene/Solr Meetup group.

Justin Mason: Links for 2013-06-15

2013-06-15T23:58:02+00:00

EU unlocks a great new source of online innovation

Today the European Parliament voted to formally agree new rules on open data – effectively making a reality of the proposal which I first put forward just over 18 months ago, and making it easier to open up huge amounts of public sector data.
Great news — wonder how it’ll affect the Ordnance Survey of Ireland?

(tags: osi mapping open-data open data europe eu neelie-kroes)
UK ISPs Secretly Start Blocking Torrent Site Proxies | TorrentFreak

The next step of cat-and-mouse. Let’s see what the pirate sites do next…
The blocking orders are intended to deter online piracy and were requested by the music industry group BPI on behalf of a variety of major labels. Thus far they’ve managed to block access to The Pirate Bay, Kat.ph, H33T and Fenopy, and preparations are being made to add many others. The effectiveness of these initial measures has been called into doubt, as they are relatively easy to bypass. For example, in response to the blockades hundreds of proxy sites popped up, allowing subscribers to reach the prohibited sites via a detour. However, as of this week these proxies are also covered by the same blocklist they aim to circumvent, without a new court ruling. The High Court orders give music industry group BPI the authority to add sites to the blocklist without oversight. Until now some small changes have been made, mostly in response to The Pirate Bay’s domain hopping endeavors, but with the latest blocklist update a whole new range of websites is being targeted.

(tags: bittorrent blocking filesharing copyright bpi piracy pirate-bay proxies fenopy kat.ph h33t filtering uk)

Nick Kew: Cui bono?

2013-06-15T22:27:28+00:00

Cicero may have popularised Cassio’s wise words “to whose benefit?“, but in our cynical times we need to refine the question: to whose expectation of benefit? Indeed, it seems implausible that the subtle distinction should have been lost on the Romans, but I certainly lack the comprehension of their language that would enable me to judge such nuances.

WordPress records show the above as the first words of a draft saved, but not published, on February 26^th 2012, following the death in Syria of distinguished journalist Marie Colvin. Who could expect to benefit from her death? Or indeed from the escalation of events both before and since: an incursion at the Turkish border, various massacres. Most recently the use of sarin gas, coming conveniently shortly after Obama had spoken of chemical weapons as a ‘red line’ that would provoke a change of policy.

The answer must surely be, someone looking to provoke Western intervention. Someone given hope of powerful backing by western rhetoric, and by events in Libya. They’ve been disappointed for a long time, but now finally it seems Obama will supply them[1] with weapons. For anyone else to engage in such gratuitously provocative yet militarily futile acts would be extraordinarily perverse. Above all, for a government with nothing to gain and everything to lose if the West were to get seriously involved (not to mention a ruthless but quietly efficient president without the vain showmanship of Saddam or Gaddafi).

Nor could you rule out someone with an even more sinister Agenda, like the CIA or Al Qaeda, or one-off maverick nutters, with whom neither ‘side’ would wish to be associated. The latter can be the ones who have the most devastating effects of all, as in the assassins of Franz Ferdinand or Yitzhak Rabin.

Can this be lost on our politicians and their advisors? Seems unlikely. I suspect much of the current rhetoric is driven by a complex case of good-cop-bad-cop desperately hoping to achieve something. Those Western politicians who really want military intervention do so for external reasons: to topple a regime with a history of the two great regional crimes of being friendly with Iran and hostile to Israel (even if Israel itself would rather have a devil-you-know relationship with a stable neighbour than a civil war)!

Could a new Western-friendly president in Iran change the situation? It’s an interesting prospect (and will probably spare Iran the kind of disturbance that followed re-election of the ‘wrong’ man last time), but I fear it’s too late to make much difference. Events in Syria have momentum. Likewise in the West: if the more gung-ho of American politicians and their backers rebuffed Khatami[2] in more peaceful times, how likely are they to change now, when it would mean some serious backing down? But at least Rohani’s probable election could serve to strengthen the hands of those favouring peace in the region including, I think, Obama himself.

Where I think the West must really bear guilt is in provoking the war in the first place. The ambiguous rhetoric and the Libyan example led rebels to suppose they’d get support if things got bad enough, but also westernised media-savvy Syrian emigrants who “spoke for” the country when it was all starting, spinners of propaganda like the “gay girl“, and doubtless others, all contributed. The contrast must surely be Bahrain, where a similar uprising was suppressed by a government that was historically more repressive than Syria’s. The obvious difference is that with no agents provocateurs or prospect of international support, Bahraini protestors cut their losses rather than escalate when the government reacted firmly to them. Bahrain didn’t get Egyptian-style democracy, but neither did it get the horrors of civil war.

[1] Who “they” may be, and whether there is a faction less guilty than the government to whom the West could supply weapons is an altogether different question. Not one I could speculate on.
[2] The Iranian president from 1997-2005, who made serious efforts to mend fences with the West but was firmly rebuffed by the US, sending a message that the West wasn’t interested and that a Western-friendly leadership was a waste of time.

Justin Mason: Links for 2013-06-14

2013-06-14T23:58:01+00:00

There’s a map for that

‘Not long ago, we began rendering 3D models on GitHub. Today we’re excited to announce the latest addition to the visualization family – geographic data. Any .geojson file in a GitHub repository will now be automatically rendered as an interactive, browsable map, annotated with your geodata.’ As this HN comment notes, https://news.ycombinator.com/item?id=5875693 — ‘I’d much rather Github cleaned up the UI for existing features than added these little flourishes that I can’t imagine even 1% of users use.’ Something is seriously wrong in how GitHub decides product direction if this kind of wankology (and that Judy-array crap) is what gets prioritised. :( (via Marc O’Morain)

(tags: via:marc github mapping maps geojson hacking product-management ui pull-requests)
Lawsuit Filed To Prove Happy Birthday Is In The Public Domain; Demands Warner Pay Back Millions Of License Fees | Techdirt

The issue [...] is that it’s just not cost effective for anyone to actually stand up and challenge Warner Music, who has strong financial incentive to pretend the copyright is still valid. Well, apparently, someone is pissed off enough to try. The creatively named Good Morning to You Productions, a documentary film company planning a film about the song Happy Birthday, has now filed a lawsuit concerning the copyright of Happy Birthday and are seeking to force Warner/Chappell to return the millions of dollars it has collected over the years. That’s going to make this an interesting case.

(tags: music copyright law via:bwalsh public-domain happy-birthday songs warner-music lawsuits)
graphite-metrics

metric collectors for various stuff not (or poorly) handled by other monitoring daemons Core of the project is a simple daemon (harvestd), which collects metric values and sends them to graphite carbon daemon (and/or other configured destinations) once per interval. Includes separate data collection components (“collectors”) for processing of: /proc/slabinfo for useful-to-watch values, not everything (configurable). /proc/vmstat and /proc/meminfo in a consistent way. /proc/stat for irq, softirq, forks. /proc/buddyinfo and /proc/pagetypeinfo (memory fragmentation). /proc/interrupts and /proc/softirqs. Cron log to produce start/finish events and duration for each job into a separate metrics, adapts jobs to metric names with regexes. Per-system-service accounting using systemd and it’s cgroups. sysstat data from sadc logs (use something like sadc -F -L -S DISK -S XDISK -S POWER 60 to have more stuff logged there) via sadf binary and it’s json export (sadf -j, supported since sysstat-10.0.something, iirc). iptables rule “hits” packet and byte counters, taken from ip{,6}tables-save, mapped via separate “table chain_name rule_no metric_name” file, which should be generated along with firewall rules (I use this script to do that).
Pretty exhaustive list of system metrics — could have some interesting ideas for Linux OS-level metrics to monitor in future.

(tags: graphite monitoring metrics unix linux ops vm iptables sysadmin)
Former NSA Boss: We Don’t Data Mine Our Giant Data Collection, We Just Ask It Questions

‘Well, that’s – no, we’re going to use it. But we’re not going to use it in the way that some people fear. You put these records, you store them, you have them. It’s kind of like, I’ve got the haystack now. And now let’s try to find the needle. And you find the needle by asking that data a question. I’m sorry to put it that way, but that’s fundamentally what happens. All right. You don’t troll through the data looking for patterns or anything like that. The data is set aside. And now I go into that data with a question that – a question that is based on articulable(ph), arguable, predicate to a terrorist nexus.’
Yep, that’s data mining.

(tags: data-mining questions haystack needle nsa usa politics privacy data-protection michael-hayden)
fastutil

fastutil extends the Java™ Collections Framework by providing type-specific maps, sets, lists and queues with a small memory footprint and fast access and insertion; provides also big (64-bit) arrays, sets and lists, and fast, practical I/O classes for binary and text files. It is free software distributed under the Apache License 2.0. It requires Java 6 or newer.
used by Facebook (along with Apache Giraph, Netty, Unsafe) to speed up “weekend Hive jobs” to “coffee breaks”. http://www.slideshare.net/nitayj/2013-0603-berlin-buzzwords

(tags: via:highscalability facebook giraph optimization java speed fastutil collections data-structures)
Big Memory, Part 4

good microbenchmarking of a bunch of Java collections; Trove, fastutil, PCJ, mahout-collections, hppc

(tags: java collections benchmarks performance speed coding data-structures optimization)

Hiranya Jayathilaka: More Reasons to Love Python - A Lesson on KISS

2013-06-14T19:03:36+00:00

Recently I've been doing some work in the area of programming language design. At one point I wanted to define a Python subset which allows only the simplest Python statements without loops, conditionals, functions, classes and a bunch of other high-level constructs. So I looked into the grammar specification of the Python language and I was astonished by its simplicity and succinctness. Click here to take a look for yourself. It's no longer than 125 lines of text, and the whole thing can be printed on one side of an A4 sheet. This is definitely one of those instances where the best design is also the simplest design. No wonder everybody loves Python.

However that's not the whole point. Having selected a suitable Python subset, I was looking into ways for implementing a simple parser for those grammar rules. I've done some work with JavaCC in the past, so I straightaway jumped into implementing a Java-based parser for the selected Python subset using JavaCC. After a few hours of coding I managed to get it working too. The next step of my project required me to do some analysis on the abstract syntax tree (AST) produced by the parser. I was looking around for some existing work that fits my requirements, and I came across Python's native ast module. I immediately realized that all those hours I spent on implementing the JavaCC-based parser is a complete waste. The ast module provides excellent support for parsing Python code and constructing ASTs. This is all you have to do parse some Python code using the ast module and obtain an AST representation of the code.

import ast

# The variable 'source' contains the Python statement to be parsed
source = 'x = y + z'
tree = ast.parse(source)

The ast module supports several modes. The default mode is exec which supports parsing a sequence of Python statements. The module also supports a special eval mode which can be used to parse simple one-liner Python statements. It turned out the eval mode supports more or less the same exact Python subset I wanted to use. So I threw away my JavaCC-based parser and wrote the following snippet of Python code to get my job done.

import ast

# The variable 'source' contains the Python statement to be parsed
source = 'x = y + z'
tree = ast.parse(source, mode='eval')

Now when it came to analyzing the AST produced by the parser, the ast module again turned out to be useful. The module provides two helper classes, namely NodeVisitor and NodeTransformer which can be used to either traverse or transform a given Python AST. To use these helper classes, we just need to extend them and implement the appropriate visit methods. There's a unique top level visit method and one visit_ method per AST node type (e.g. visit_Str, visit_Num, visit_BoolOp etc.). Here's an example NodeVisitor implementation, that flattens a given Python AST into a list.

class NodeEnumerator(ast.NodeVisitor):
  def get_node_list(self, tree):
    self.nodes = []
    self.visit(tree)
    return self.nodes

  def visit(self, node):
    self.generic_visit(node)
    self.nodes.append(node)

These helper classes can be used to do virtually anything with a given AST. If you want you can even implement a Python interpreter in Python using this approach. In my case I'm running some search and isomorphism detection algorithms on the Python AST's.

So once again I've been pleasantly surprised and deeply impressed by the simplicity and richness of Python. It looks like the designers of Python have thought of everything. Kudos to Python aside, this whole experience taught me to always looks for existing, simple solutions before doing it in my own complicated way. It actually reminds me of the good old KISS principle - "Keep It Simple, Stupid".

Chip Childers: The CloudStack Collab Conference is Almost Here

2013-06-14T18:14:50+00:00

We're getting close to the CloudStack Collaboration Conference for 2013, being held on June 23 through 25 in Santa Clara, CA. If you haven't registered, there's still time (but not much), because tickets are really going fast right now. Go here: http://ccc13.eventbrite.com/

We've got a great schedule lined up for the event, with keynotes from Gene Kim and Adrian Cockcroft (as well as myself). Talks are scheduled for the 24th and 25th, striking a great balance between some fantastic user stories, more development focused discussions, and presentations about some of the other projects in the Apache CloudStack ecosystem. We'll also be doing lightning talks over lunch, so that people have a chance to share in a shorter and more infomal setting. Oh, and of course, a hack-a-thon on Sunday the 23rd. If you're new to Apache CloudStack, ShapeBlue is actually offering a CloudStack Bootcamp training program on the 22nd and 23rd.

I'm looking forward to seeing others from the communty at the event, especially those that I have never met in person.

Bryan Pendleton: I'm starting to get nervous about losing Google Reader

2013-06-13T21:49:18+00:00

We're down to the last two weeks of availability of Google Reader.

Like tens of thousands of you, I'm trying to figure out what to switch to.

I tried The Old Reader, and it looks pretty nice, but it hasn't been very reliable. I frequently get the pictures of kittens, with a polite message apologizing for their inability to show me actual data. So they seem to be polite, and hard-working, but over-whelmed.

I tried NewsBlur, and it looks appealing, but they won't let me add even a single feed without giving them my credit card. It's not all that expensive ($2/month), but I was really hoping I'd get to try it for free and see if it worked or not first.

I guess maybe the next ones I'll try are Feedly and NetVibes.

Am I nuts? Is this working better for anyone else? What is everyone switching to?

Sergey Beryozkin: JAX-RS 2.0 is out !

2013-06-13T21:14:45+00:00

You may have already heard JAX-RS 2.0 (JSR-339) has been released. This is a very good news for Java developers building their RESTful HTTP applications and here are the reasons why.

The 2.0 API offers a lot of new enhancements on top of already very capable JAX-RS 1.1 (JSR-311) API and spec. It has really been a very serious push to the next level across all the API and the specification text. And what is really good is that the community can now rest assured: the JAX-RS effort started by Paul Sandoz and Marc Hadley is very alive and is led by a new super team, Marek Potociar and Santiago Pericas-Geertsen who worked extremely hard to get JAX-RS 2.0 out.

As I said the new API offers a lot of enhancements: client and server filters and interceptors, new Client API, client and server asynchronous invocation support, a lot of new context classes, new exception classes, a lot of API updates. A lot of new things to try and learn, but what I personally like most is the new asynchronous API and the fact that JAX-RS 1.1 Response class can be reused by Client API - it seems like a minor thing but IMHO it's one of the major points.

I've enjoyed taking part in the JAX-RS 2.0 development process. Talking to all the participants, 'fighting to death' on the subject of very trivial issues, running never ending threads - it has all happened and it has been great :-). As every developer out there knows sometimes one has to forget about the disappointment of having your great point of view rejected :-), and move on with the team toward making something good happen - this is what this spec effort has been about.

Talking to one of my former team leads, Bill, has been super too :-). As a side note, a lot of API enhancements have their origin in the work Bill and his team did.

JAX-RS 2.0 is out and about to take over and fly high. It is only a start, MVC support, some new useful features will hopefully and likely be in in the next major releases. Is it perfect ? Probably not - I can name few API and spec features I'm still not feeling exactly happy about, but overall JAX-RS 2.0 is great.
In Apache CXF we are going to have JAX-RS 2.0 completely implemented by the end of the year. As it happens 2.0 server-side API is already completely implemented on the trunk. Some of the features like the integration with Bean Valdation API is required by the EE profile only but I think we can wire that into CXF too.
It is also validated against the early TCK 2.0 build (with Andrei helping a lot).

Client API is 50% done - CXF WebClient API has been enhanced to support new Client interceptors and filters, Response and even Asynchronous Invocations (with thanks to Dan). Still some work to do on the actual new Client API. We are getting there.

So try JAX-RS 2.0 API and enjoy !

Matt Raible: Life Update

2013-06-13T16:34:26+00:00

For the past several years, I've written "farewell to ski season" posts at the end of April. This year, I intended to as well, but I never knew it was going to end!

A lot's been going on since I last wrote here. First of all, the end of the ski season was fantastic! The last week in Winter Park was the best skiing of the year. The first (Tuesday) afternoon, I started skiing around 1pm and it snowed all afternoon - resulting in several inches by the end of the day. The next day was smooth and empty, followed by a day of deep powder and knee-deep runs down Eagle Wind. I had one of the best runs of my life that day.

While the snow was great at the end (and far better than last year), I didn't get as many days as I'd like. Only 22 this year. Most of them with Trish, Abbie and Jack. Next year, we'll be getting Abbie a Colorado 5th Grade Free Ski Pass and trying to hit all the ski resorts in the state. Below are some of my favorite pictures from this year's season - can't wait for next year!

The Bus Project
The last time I wrote about The Bus, it was in October. Since then, its rear suspension and wheels/times have been installed. In February, I bought a brand-new 911 racing engine and it's been installed as well.

There's still quite a bit of suspension, drivetrain and steering work to do, but it's in good hands. Last week, I removed it from Motorworks Restorations and I've started finishing it myself. I don't have welding skills and equipment, so I'm contracting that out, as I will most of the finishing touches. I'm also thinking of writing a letter to one of those Body Shop shows on TV and seeing if I can get them to finish it.

New Gig
If you've seen my LinkedIn Profile lately, you'll know I was looking for a new gig. I found one in mid-May and started consulting at Travelport in June. It's a nice 11-mile bike ride (uphill on the way there, downhill coming home) that I've thoroughly enjoyed. Takes me about an hour each way and is much better than sitting in traffic for 45 minutes. The project is interesting and fun to work on, and it ends in a couple weeks.

The Wedding and Honeymoon
In July, we're heading up to The Cabin in Montana to vacation and get ready for our wedding. We'll be enjoying the outdoors (rafting, biking, fishing), visiting with my folks and getting ready for the big day. After that, we'll be heading on a whirlwind honeymoon that takes us from Ireland to Italy to Thailand to Fiji. We're very much looking forward to the whole adventure.

And with that, I must go. It's Craft Cruisers night and there's beer to drink.

Edward J. Yoon: WWW에 대한 잡설

2013-06-13T10:45:12+00:00

WWW에 하이퍼텍스트의 결합으로 HTML이 탄생하고 정보의 바다, 정보도서관이 생성되니 어느 순간 검색엔진이 출현하였다. 쇼핑몰들이 대거 출현하더니 그 다음으로 사용자들의 적극참여와 함께 발전해나가는 오늘날의 위키, 블로그와 소셜 서비스가 출현하는 순서다.

그 중간 과정에서 똑똑한척하던 많은 이들이 미래의 웹에 대해 헛다리짚은 것이 있겠으니 그게 시만틱웹과 인공지능. 구글이라면 왠지 가능할 것도 같지만 향후 (가까운) 미래의 웹 발전 방향에 대한 내 생각은 전혀 다름.

IT 기술의 트렌드도 모두 필요에 의해 시작되고 발전하며 변화하지. 아주 자연스럽게, 과하게 우아한 연구 개발들은 사라질 수 밖에.

Shane Curcuru: ALERT: I’m scheduling my midlife crisis next week

2013-06-12T20:56:47+00:00

Dear Friends,

A number of factors in my life lately have led to a whole lot of stress – above and beyond normal life – for a while now. Therefore, in part to deal with these issues, I am hereby declaring a midlife crisis. I plan to start it in exactly one week [1], at lunchtime, on Friday June 21st, and stretching throughout the summer months. I’m sure you all have many questions, so let me try to address some of them now.

Shane! What’s wrong, Shane?

I’ve decided to have a midlife crisis. Isn’t that the responsible thing to do? I figure if I schedule some of it out, it’ll be much more convenient for everyone.

Are you OK?

Yes, in the big picture, I’m still better off than most humans on the planet: I have a wonderful wife, daughter and cats; a comfy home; and a paying job. So color me #FirstWorldProblems for sure. But…

But what? What’s been going on?

The Archduke Franz Ferdinand moment was probably the death of my father, in January of this year. Dealing with both the personal issues and especially dealing with the paperwork and other extended family estate issues was far more than I expected, and is continuing. This also led to a number of other stresses and health issues for me, which of course got me worried about being stressed… Plus, my father was a hoarder, and the responsibility of cleaning out his house(es) is a huge undertaking that’s not even half done.

Coupling this with both the hectic normal life of a family with a young school child, two working parents with busy jobs, plus my significant (and growing) volunteer responsibilities at the ASF, has just been more than is comfortable to deal with.

So what are you doing about it?

Well, right now I’m writing this witty and informative blog post about it, just for you!

(ahem) No, really.

Well, I figured since I’m of that age anyway, I might as well schedule a midlife crisis. See, one of the skills I’ve always thought I should work on is better time management, so I figure if I can schedule a midlife crisis I’d already be improving myself.

Hence, to have a proper midlife crisis, I am taking an unpaid leave of absence from my job. Thus while I may be without regular income (but see: #FirstWorldProblems), I will have ample time to do all those crazy and fun things that go with a traditional midlife crisis. And since I won’t have a 9-to-5 (ha!) for a while, I should be much less worried about “getting things done” and just relax.

Cool – so what are you planning to do?

Well, it may sound trite, but I plan to buy a new(er) car. While I may love my current BMW very dearly, he’s finally started to succumb to the dreaded E36 rust monster. Be on the lookout for an E90 sedan w/manual & sunroof for me.

I also plan to enjoy the summer. While I’ve had sabbaticals before (thank you Lotus!), I’m really looking forward to having a whole 12 weeks of no job. I’ll do some cool stuff with my family, we’ll go visit family & friends a bunch, and I’ll do some important – or not so important, but fun – projects at home. Perhaps finishing my retro man cave, or finally organizing my entire digital life.

That sounds good – but will it help?

Not sure – I’ve never had 3 months off from my job since I started working at Lotus almost 2X years ago. But I also hope to work on a bunch of those things that I keep wanting to really do, but never seem to complete. I’m at the point in life where it’s clear to me that having more available time is more important right now than, say, income. And, #FirstWorldProblems, I am lucky enough in my career to be able to take a temporary leave from work (thanks, manager!).

Good for you. What does this have to do with me?

Well, one of the most important uses of additional free time that I can think of is to spend more time with friends! So one of the (again, stereotypical) things I hope to do is a few guy trips or dinners out with friends. Movies, good dinners, a drive somewhere, some card games, a trip to MV, or a recap of my 10th anniversary bachelor party would all be fun.

Likewise, it would be great to see families and friends, and local friends from school, and so on. We have a few weeks scheduled already, but otherwise plenty of time in the summer available now. So that’s what it does with you – I’d love to see all you friends more!

In any case, wrapping up things here at work will be quite busy until then, but do send me an email if you have cool ideas of stuff to do this summer.

Thanks for asking those great questions, by the way. That’s just what I needed to feel like I have a good start to my midlife crisis.

[1] BHD and the astute readers of my blog will of course note the discrepancy, which I will note is likely a result of stress getting the date wrong. That, or I’m just trying to be funny.

Rob Weir: The Power of Brand and the Power of Product, Part 2

2013-06-12T17:55:05+00:00

In Part 1 of this series we looked at a model of product adoption and market share that had a special and valuable property: the parameters of the model could be derived from a single survey question, e.g.:

“What is your awareness with the hand cream called Whizzo-Soft?”

A. I have never heard of it.

B. I have heard of it but I have never tried it.

C. I have tried it once.

D. I use it sometimes.

E. I use it regularly.

Given N responses to that survey questions you can derive the factors in the model by simple math:

Customer Awareness = 1 – A/N
Customer Motivation = (C + D + E) / (N -A)
Customer Satisfaction = (D + E)/(N – A – B)
Market Share = Customer Awareness * Customer Motivation * Customer Satisfaction

So let’s take a look at how this can be used in practice, taking the leading open source office productivity editor, OpenOffice, and the lesser known LibreOffice fork, as examples.

As mentioned in Part 1, the execution of the survey is critical here. Without a proper, random survey of the market, the results will not be accurate. In particular a survey of your current users will not work, since one of your goals is to find out what proportion of users are not familiar with your product.

So in this case I used Google’s new Consumer Survey service which uses sampling and post-stratification weighting to match the target population, which in this case was the US internet population. In other words, the survey is weighted to reflect the population demographics, for age, sex, region of the country, urban versus rural, income, etc. I did this survey in a personal capacity for my own interest. The Standard Disclaimer applies.

They survey question (and responses were):

What is your familiarity with the software application called “OpenOffice”?

I have never heard of it
I am aware of it but have never used it
I have tried it once
I use it only sometimes
I use it on a regular basis

With 1502 responses, the results were:

I have never heard of it	72.4%
I am aware of it but have never used it	9.3%
I have tried it once	5.7%
I use it only sometimes	5.9%
I use it on a regular basis	6.6%

And then with some simple arithmetic we have:

Customer Awareness	27.6%
Customer Motivation	65.9%
Customer Satisfaction	68.7%
Market Share	12.5%

What does that mean? In plain English:

Around 1/4 of US internet users have heard of the OpenOffice software application. That is the brand recognition.
Of those who have heard of OpenOffice, around 2/3 of them were sufficiently motivated to try the software.
And of those who tried OpenOffice 69% were sufficiently satisfied with the software that they continue to use it.
Overall, 1/8 of the surveyed population uses OpenOffice sometimes or regularly.

The absolute numbers are tricky to interpret in isolation. More interesting is to look at the numbers over time. The same survey question, with the same methodology was also given last September. The results and the change are in the following table, with changes having statistical significance (90% confidence level) emphasized in bold.

OpenOffice	September 2012	April 2013	Change
Customer Awareness	24.3%	27.6%	14% growth
Customer Motivation	63.0%	65.9%	5% growth
Customer Satisfaction	70.6%	68.7%	3% decline
Market Share	10.8%	12.5%	16% growth

The Apache OpenOffice project should be gratified that their efforts have paid off, and awareness of the product is increasing, as well as market share. This goes contrary to some loudly expressed concerns that the OpenOffice brand would languish at Apache. Clearly this is not so. The brand is growing, as well as the market share.

Since these factors are multiplicative, an increase in any one of them, or any combination of them, will grow the market share. But it is probably easiest to grow the factor that is smallest today. So looking to the future, increasing the awareness of the existence of OpenOffice would give the “biggest bang for the buck”.

For an entirely different view we can look at the same survey question and methodology, administered at the same times, only substituting the product name “LibreOffice” for “OpenOffice”. Again, statistically significant changes are shown in bold.

LibreOffice	September 2012	April 2013	Change
Customer Awareness	10.7%	9.9%	7% decline
Customer Motivation	53.3%	66.7%	27% growth
Customer Satisfaction	73.7%	59.7%	19% decline
Market Share	4.2%	4.0%	5% decline

The brand recognition is not growing and is stuck at 10%. The fact that in its third year of product availability the LibreOffice brand recognition has plateaued (if not declined) should be a concern.

But the more interesting thing here is the large increase in users trying LibreOffice (Motivation) offset by the large decrease in users who continue to use the product (Satisfaction). What does this mean? Only the LibreOffice folks can say for certain, but this pattern is exactly what one would expect from a product where marketing has got ahead of quality. It is like a movie that previews well, but suffers from bad reviews and poor sales after the first weekend. Product development aims to make products that users want. And marketing persuades users to try the product. But where there is a disconnect between the two, where the product is not fulfilling the needs of those to whom it is being marketed, or (the same thing really) the product is being marketed to unsuitable users, this is what you see.

I should note that LibreOffice supporters like to blame their lack of success on not having the OpenOffice brand. Yes, having a familiar brand is a nice thing to have, but the drop in Satisfaction for those trying LibreOffice is not a brand issue, since it is entirely among those who are already familiar with the LibreOffice brand. Satisfaction is an attribute of the product, not due to brand.

Also, we can compare the metrics across products. When we look at the most recent data OpenOffice clearly has an enormous lead in name recognition and market share, but also a large lead in Satisfaction. 69% of those who tried OpenOffice remained users, compared to 60% for those who attempted to use LibreOffice. Keep your users satisfied and it is hard to go wrong.

Finally, and to reiterate up what I wrote earlier in my Scarcity Fallacy post, when you consider the position of Microsoft Office in this market, both products have a relatively small presence, with ample of room to grow, at Microsoft’s expense. This is a great area to advance the cause of open source software, in a product category that almost every user needs. There is no shortage of opportunity here, only a shortage of imagination. Imagine if we combined the stability/quality and brand recognition of Apache OpenOffice with the enthusiastic marketing team of LibreOffice? (Combine our 50 million downloads with their 50 million press releases) What if we combined the disciplined development approach of OpenOffice with LibreOffice ‘s talented developers? Imagine what we could do?

Let’s admit it. LibreOffice has plateaued. They have their Linux desktop users, all 3% of the market that runs Linux on the desktop. This market share was not earned. These are not users that they won over. These are users they got via the control their corporate sponsors have over Linux distributions. They flipped a bit and instantly had that market share. But their sponsors are Linux vendors that have little motivation to reach beyond that niche market. (They certainly have little success doing so). The opportunity for growth is not on the Linux desktop, unless the goal is to merely be a small fish in an even smaller pond. Of course, LibreOffice could continue, and languish indefinitely as a pet project of a handful of Linux developers. Or they could work with us at Apache, and satisfy the Linux users, but do so very much more as well. This would also be a cost savings for LibreOffice’s corporate sponsors, no small factor in a world of declining PC sales. The choice now, as it always has been, is theirs.

Bryan Pendleton: I continue to be baffled by e-book pricing

2013-06-12T16:13:13+00:00

I felt like I wanted to re-read Watership Down; it's been 35 years since I read it and I'd like to re-explore it.

On Amazon, I can find:

New ("mass market") paperback editions for $6.50
Used paperback editions for less than $1.00 (+shipping)
New paperback edition from Amazon is $13.00
Kindle edition is $11.00

Does anybody actually pay $11.00 for the Kindle edition of a book you can get for half that price?

I thought about just getting the pirated version from the Internet, but decided instead to buy a used copy for $3.00.

I find the whole Kindle thing bizarre. It's such a gorgeous device, but in practice I'm buying almost no content for it.

As the kids say, "just saying."

Adrian Sutton: Importing an ant buildfile multiple times

2013-06-11T23:49:30+00:00

According to the ant documentation:

It is possible to include the same file more than once by using different prefixes, it is not possible to import the same file more than once.

Unfortunately, it turns out this isn’t quite true. In most cases, ant will indeed ignore a request to import a file for the second time, so:

<import>
   <file name="utils.xml"/>
   <file name="utils.xml"/>
</import>

will only import utils.xml once. This is true even if there’s a chain of files being imported (so A imports B and C, then B imports C as well, C will only be imported once).

The problem arises if you’re importing the file via two different methods, you might wind up having it actually be imported twice. In my case I was loading build files from a zip archive (why is a long story that I should write up some time) and those build files references other files also within the zip file. So my main build file had:

<import>
  <javaresource classpath="bootstrap.zip" 
                     name="basic-build.xml"/>
</import>

Then basic-build.xml included another utils file from the same zip. Being the well-coded ant file that it is, basic-build.xml doesn’t depend on knowing where it’s actually located, so it imports the utils file via:

<import>
  <url baseurl=${ant.file.basic-build}" relativePath="ivy-utils.xml"/>
</import>

This all works just fine. Then there came a requirement in our main build file to use one of the macros from ivy-utils.xml to set a version number before basic-build.xml was imported. So the main build file wound up with something like:

<import>
 <javaresource classpath="bootstrap.zip"
                    name="ivy-utils.xml"/>
</import>
<!-- Do some stuff -->
<import>
 <javaresource classpath="bootstrap.zip" 
                    name="basic-build.xml"/>
</import>

Suddenly the build started failing because ivy-utils.xml was being executed twice and attempting to redefine the ivy configuration which was explicitly disallowed.

Looking at the ant -debug output clearly showed the ivy-utils.xml was being imported twice and not skipped, and even that it came from exactly the same URL. It appears that resolving the same file using two different mechanisms (javaresource and url in this case) caused ant to believe they were different files.

The answer was fairly straight-forward, use url resources consistently when importing the files:

<import>
 <url url="jar:file:${ant.file}/bootstrap.zip!/ivy-utils.xml"/>
</import>
<!-- Do some stuff -->
<import>
 <url url="jar:file:${ant.file}/bootstrap.zip!/basic-build.xml"/>
</import>

Building a jar: URL like this is a little unpleasant but not unreasonable. Strictly we could have kept importing basic-build.xml using the javaresource approach but being consistent will save surprises in the future.

Ben Laurie: Rice Pudding

2013-06-11T15:32:13+00:00

For almost all of my life, I have hated rice pudding. But recently my wife made some (up to now, I’ve only seen her eat it from a tin – disgusting stuff). And it was awesome. So now I keep making rice pudding.

I recently had occasion to cook it for a lot of people. Here’s the recipe.

500g pudding rice
250g demarara sugar
1l whole milk
flavourings (see below)
ground nutmeg

Flavourings: there’s probably a lot you can do, but the two I did most recently were (choose one):

a) Zest of two oranges (I like to peel them with a potato peeler and then finely slice the peel) plus ground seeds from about 20-30 cardamom pods (pods removed).

b) Zest of two limes and two lemons (as above).

Put nearly all ingredients in a flat oven dish, should be about 1.5″ deep and mix. Dust the top with nutmeg. Cook at gas mark 3 for 2 hours.

This should give you a nearly completely dried out rice pudding, with just a touch of moistness and a strong flavour. Which is how I like it.

Serve with double cream. Should feed 12 generously.

What? I didn’t say it was slimming!

Nick Kew: New laptop

2013-06-10T00:06:55+00:00

After 7 years heavy use, my old macbook is showing its age. The battery has long been knackered: just about adequate to move the machine from one room to another and plug it in to the mains. The backlight failed last year and needs nursing to keep it working. And now the battery has reached a point where it powers down if I just accidentally knock the power lead out of place for half a second.

It’s been a good little machine and served its purpose: a laptop with a Unix-family OS and hardware that just worked. Plus a fantastic display quality that made a desktop substitute of the 13″ screen. It’s not been trouble-free: I had to replace the disc a while back, but on the whole it’s been great. But now with two major expensive-to-repair faults, I guess it’s time to look around the market again, and take advantage of other advances, notably further reductions in size and weight since 2006.

I don’t think I’ll go for another Macbook. My experiences with more recent macbooks have been rather less positive than the old one, while at the same time I’d expect there to be a much wider range of laptops where the more ‘challenging’ things of 2006 – like ACPI and builtin wifi – just work, without hassle.

So what’s a good laptop for Linux[1], or even with Linux preinstalled? Another 13″ screen will suffice provided the display is of comparable quality to the old Mac, and I’d love it to be genuinely small and light with a good battery life. That probably implies ‘ultrabook’. And since I’ll be doing lots of work with the GNU toolchain – which can write hundreds of thousands of tempfiles in a typical build – I can’t alas go for an SSD-only machine.

A bit of preliminary poking around suggests cheapo ultrabooks from Asus or Acer as good candidates with positive experiences from Linux users, and Lenovo and Toshiba as labels to avoid in the ultrabook space. Comments solicited from readers who know more than I: will a £500-ish ultrabook be a decent working machine, or is it likely to be as shoddy and useless as the Dell I had before the macbook? Anything in particular to look out for? Any further suggestions?

[1] Or I’d consider other-*X if someone convinced me it would be hassle-free on an appropriate piece of hardware.

Michael McCandless: Build your own finite state transducer

2013-06-09T15:01:31+00:00

Have you always wanted your very own Lucene finite state transducer (FST) but you couldn't figure out how to use Lucene's crazy APIs?

Then today is your lucky day! I just built a simple web application that creates an FST from the input/output strings that you enter.

If you just want a finite state automaton (no outputs) then enter only inputs, such as this example:

If all of your outputs are non-negative integers then the FST will use numeric outputs, where you sum up the outputs as you traverse a path to get the final output:

Finally, if the outputs are non-numeric then they are treated as strings, in which case you concatenate as you traverse the path:

The red arcs are the ones with the NEXT optimization: these arcs do not store a pointer to a node because their to-node is the very next node in the FST. This is a good optimization: it generally results in a large reduction of the FST size. The bolded arcs tell you the next node is final; this is most interesting when a prefix of another input is accepted, such as this example:

Here the "r" arc is bolded, telling you that "star" is accepted. Furthermore, that node following the "r" arc has a final output, telling you the overall output for "star" is "abc".

The web app is a simple Python WSGI app; source code is here. It invokes a simple Java tool as a subprocess; source code (including generics violations!) is here.

Adrien Grand: Versatile sorting

2013-06-08T22:15:00+00:00

Sorted data sets are very useful since they make a lot of things easier:

checking for duplicates,
computing frequencies (the number of times each unique element appears),
compression (thanks to delta encoding and bit-packing for example),
searching thanks to binary search,
…

Java provides the ability to sort arrays and lists thanks to Arrays.sort and Collections.sort. These methods are flexible enough to accept custom comparators, so that you can define which sort order to use depending on your needs, but you don’t have the choice on the sorting algorithm although they can have very different speed and memory characteristics. Here is how the Oracle JVM 1.7 sorts data:

Arrays.sort(Object[]) uses TimSort,
Arrays.sort on native arrays uses a dual-pivot quicksort,
Collections.sort dumps the list into an Object[] array, sorts the array with TimSort, and then re-adds elements to the list from the array.

Although these APIs serve most use-cases, sometimes you would want to have more control over the implementation, for example:

To sort parallel arrays: imagine you have an array of objects and an array of floats, where the float at offset i is the score of the object at the same offset in the other array. Now you want to sort objects by their score. It is doable by writing a list view on top of these two arrays, but since Collections.sort dumps data into an Object[] array, this would be very memory intensive.
To avoid useless object allocations: to sort a random-access list (such as ArrayList, but not LinkedList), there is no need to dump all your elements into an array, you could instead sort the list in place and save memory.
To better fit your data: if your data is “almost” sorted, there is a good chance that TimSort would perform faster than Quicksort.
To better fit your constraints: if you want to sort a huge array which occupies a large part of your heap, then TimSort might not be the best algorithm since it requires up to n/2 temporary slots. You could instead use an in-place sorting algorithm.
To better reuse memory: say you want to sort 100,000 small Object[] arrays. For every array, Java’s Arrays.sort will create a new temporary array (TimSort needs temporary storage to perform merges). By having more control over the sorting implementation, the temporary storage could be reused.

In order to have more control over sorting, Lucene initially imported CGlib’s SorterTemplate and improved it over time. But then arose the need to use TimSort to sort partially-sorted data, and it was very hard to fold this sorting algorithm into SorterTemplate since this sorting algorithm requires temporary storage (on the contrary to the quicksort and in-place merge-sort implemented by SorterTemplate). This is why I started a small GitHub project to refactor SorterTemplate so that it can support more sorting algorithms:

a modified TimSort, which allows you to configure the amount of temporary storage that can be allocated,
Merge-sort, with configurable memory overhead similarly to TimSort,
Introspective sort, which is essentially an improved Quicksort,
Heap-sort, on both binary and ternay heaps.

For example, here is some code which sorts two parallel arrays using introspective sort:

These new classes are now being used in Lucene, but I think it could be very useful to other projects, so feel free to use them! Feedback is very welcome.

Bruce Snyder: Remove/Reinstall of MacPorts

2013-06-07T23:24:52+00:00

After trying to upgrade Mercurial to the latest version via MacPorts, I ran into some compilation issues with a couple dependencies that I could not resolve at all. After realizing that there were dependency resolution issues amongst different versions of dependencies for different ports, I decided to remove and reinstall MacPorts from scratch. This provided a clean slate and everything worked without a problem, though I was annoyed that I had to take this drastic action and spend the time to get MacPorts working again.

Bash Completion For Git

One thing that I ran into was another change to the bash completion and prompt for git. As I mentioned in a previous post about this topic, the location of the location for the git-prompt.sh file had changed but this time I discovered that I had to source both the git-completion.bash file and the git-prompt.sh file to get things working. Below is what worked for me.


if [ -f /opt/local/share/git-core/contrib/completion/git-completion.bash ]; then
  . /opt/local/share/git-core/contrib/completion/git-completion.bash
fi

if [ -f /opt/local/share/git-core/contrib/completion/git-prompt.sh ]; then
  . /opt/local/share/git-core/contrib/completion/git-prompt.sh
fi

Bash Version 4.x

As I was reinstalling ports I noticed some logging that flew by in the terminal stating something about a requirement to use Bash 4 for the bash completion functionality to work properly. So I decided to utilize the newer version of Bash installed by MacPorts and here's how:

Add the path to the new bash (/opt/local/bin/bash) to /etc/shells
Use the chsh utility to change the shell to the new bash: chsh -s /opt/local/bin/bash
After closing the terminal and opening a new one, check the bash version using either of these methods:
```
$ echo $BASH_VERSION
4.2.42(2)-release
```

As long as the version displayed is the newer version from MacPorts, you should be ready to go.

Just for my own sake and in case I need it in the future, I'm going to list a number of items that I had to reinstall:


$ sudo port install git-core +bash_completion +doc +svn
$ sudo port install tig 
$ sudo port install mercurial +bash_completion
$ sudo port install gnupg curl wget ncftp tree pstree watch screen proctools mtr cowsay
$ sudo port install postgresql90-server

For more info about installing and configuring PostgreSQL, see my post titled Installing PostgreSQL 9.0 on Mac OS X 10.6.8 via MacPorts

Tim Bish: Apache.NMS.ActiveMQ v1.6.0 Released

2013-06-07T18:35:26+00:00

After almost a year since the last release the Apache.NMS.ActiveMQ 1.6.0 Release bundle is now available. We spent a lot of time on this release adding in new features and fixing bugs. You'll find many features added that were previously only available in the Java client such as optimized acknowledge and priority backups for the Failover Transport.

New in this release:

* Adds support for non-blocking redelivery.
* Supports priority backups for Failover transport.
* Supports an Optimized Ack mode for MessageConsumers.
* Adds Message Audits to prevent duplicate deliveries on Failover.
* Implements the ISession Recover method.
* Adds support for purging locally created temp destinations.
* Properly handles cluster client re-balancing in Failover Transport.
* Plus a lot more fixes and stability improvements.

This release is based on the Apache.NMS API v1.6.0 and runs on .NET frameworks 2.0+ and Mono 2.0+.

The binaries and source bundles for the Release Candidate can be found here:

The list of issues resolved is here:

There are release binaries and debug binaries (with PDBs) provided. The binary ZIP files include pre-built assemblies for all of the platforms that are supported.

Bruce Snyder: Installing PostgreSQL 9.0 on Mac OS X 10.6.8 via MacPorts

2013-06-07T17:13:18+00:00

While installing PostgreSQL 9.0 on Mac OS X again, I had to figure out all these steps yet again. So I'm documenting this process for my own sake because I have been through this now twice on two computers recently, having to hunt down all of these commands each time. I'm hopeful that this will help others as well.

Use the MacPorts command port to install PostgreSQL 9.0, then create and own a data directory and a logs directory:


$ sudo port install postgresql90-server
$ sudo mkdir -p /opt/local/var/db/postgresql90/defaultdb
$ sudo chown -R postgres:postgres /opt/local/var/db/postgresql90
$ sudo mkdir -p /opt/local/var/log/postgresql90
$ sudo chown -R postgres:postgres /opt/local/var/log/postgresql90

Now you need to initialize the database using the data directory that was created above:


$ sudo -u postgres /opt/local/lib/postgresql90/bin/initdb -D /opt/local/var/db/postgresql90/defaultdb

I prefer to change the postgres user's shell to bash:


$ sudo dscl . -create /Users/postgres UserShell /bin/bash

View the postgres user account just to make sure it all looks OK:


$ dscl . -read /Users/postgres
AppleMetaNodeLocation: /Local/Default
GeneratedUID: 5B38F583-CBBF-4082-A32D-C17947394A27
NFSHomeDirectory: /opt/local/var/db/postgresql90
Password: *
PrimaryGroupID: 501
RealName:
 PostgreSQL-90 Server
RecordName: postgres
RecordType: dsRecTypeStandard:Users
UniqueID: 502
UserShell: /bin/bash

Also check the postgres group:


$ dscl . -read /Groups/postgres
AppleMetaNodeLocation: /Local/Default
GeneratedUID: 715FEB22-D0F1-443F-BC93-55896210DB44
Password: *
PrimaryGroupID: 501
RealName: postgres
RecordName: postgres
RecordType: dsRecTypeStandard:Groups

Now edit the pg_hba.conf file to add the appropriate permissions:


# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust

This should allow you to connect easily using the psql utility.

I prefer to use the pg_ctl command to start and stop PostgreSQL. This is simply because I learned PostgreSQL on Linux and that's all there was. To prevent having to retype the full command every time I want to start or stop the database, create a start script and a stop script in the postgres user's home directory named pg_start and pg_stop. Below are the contents of the pg_start file. Make sure to create these files as the postgres user in the home directory:


$ sudo su - postgres 
$ vim ./pg_start 
#!/bin/sh 
/opt/local/lib/postgresql90/bin/pg_ctl -D /opt/local/var/db/postgresql90/defaultdb -l /opt/local/var/log/postgresql90/postgres.log start &

Below are the contents of the pg_stop file:


$ vim ./pg_stop 
#!/bin/sh 
/opt/local/lib/postgresql90/bin/pg_ctl -D /opt/local/var/db/postgresql90/defaultdb -l /opt/local/var/log/postgresql90/postgres.log stop

Don't forget to make them executable:


$ chmod +x ./pg_start 
$ chmod +x ./pg_stop

(There is a Mac OS X way of starting PostgreSQL using launchctl but I don't tend to use that because I'm used to the standard pg_ctl command.)

Now use the pg_start script to start up PostgreSQL. Execute it as the postgres user (sudo su - postgres) I tend to cat the log file just to make sure it's running correctly:


$ ./pg_start 
$ server starting

$ cat ../../log/postgresql90/postgres.log 
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

Looks good so we'll create my user:


$ /opt/local/lib/postgresql90/bin/createuser bsnyder
Shall the new role be a superuser? (y/n) y

Because I made my user a superuser, I can create my own db schema, so log out of the postgres user account and back to my own account first:


$ exit
$ /opt/local/lib/postgresql90/bin/createdb
CREATE DATABASE

The createdb command automatically uses my username as the schema name.

The only thing left to do before starting up the database is edit your ~/.profile or ~/.bash_profile to put the path to the PosgreSQL bin directory into the PATH:


export PATH=/opt/local/lib/postgresql90/bin:$PATH

Now just log into the PostgreSQL server using psql to make sure we're ready to roll:


$ psql
-bash: psql: command not found
bsnyder@skunk [darwin](DARWIN-1527) $ /opt/local/lib/postgresql90/bin/psql 
psql (9.0.4)
Type "help" for help.

bsnyder=# select version();
                                                                 version                                                                  
------------------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.0.4 on x86_64-apple-darwin10.8.0, compiled by GCC i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664), 64-bit
(1 row)

bsnyder=#

And we're good to go!

UPDATE: For those who want a nice seamless experience stopping/starting Postgres via a Mac OS X preference pane, check out John Wang's PGPane.

Bruce Snyder: MacPorts Upgrade Changes Bash Completion

2013-06-07T16:06:41+00:00

After upgrading Macports recently to version xxx, I started seeing the following error in the terminal when the .bash_profile is loaded:

__git_ps1: command not found

Upon investigation, I discovered that the bash completion for git has been split into more than one file. Previously, all that was needed in the .bash_profile was to source the single file for git like so:


if [ -f /opt/local/etc/bash_completion ]; then
  . /opt/local/etc/bash_completion
fi

Since the update to Macports, now I also need to source the second file like so:


if [ -f /opt/local/share/git-core/contrib/completion/git-prompt.sh ]; then
  . /opt/local/share/git-core/contrib/completion/git-prompt.sh
fi

After closing/creating the terminal tab, the error is gone.

Marco Di Sabatino Di Diodoro: How to customize Apache Syncope console

2013-06-06T08:43:00+00:00

David Reid: Variable Pitch Take 2

2013-06-05T22:38:48+00:00

It’s fair to say that when I started the Variable Pitch website I didn’t know a lot about the energy market or the data that would be useful and available. I had an idea and some data and I started trying to make it available. The site started with just scottish stations but soon I had people asking me about english, welsh and northern irish stations, so I expanded to also cover those countries.

It wasn’t just the coverage that expanded. While I started out focussing on Renewable Obligation certificates (ROC) and their value, I was soon being asked about other aspects of finance related to renewable energy. The Ofgem data provided a way of viewing the output for stations but my early focus on ROC was soon shown to be an error as other financial schemes such as the Feed In Tariff needed to be considered. Then there was the selling of electricity that I hadn’t thought about and the constraint payments made were another area that I had not originally anticipated tracking but something I added to the site.

As with all projects that grow in such an organic manner my initial design of data structures turned out to be totally inadequate. I added the pieces I needed but over time it grew arms and legs on it’s arms and legs. It worked, but adding things was getting harder and harder. Simple maintenance was proving a challenge. Developing the site was also getting harder.

As a result of everything above I finally decided that the time was right to bite the bullet and use the my additional knowledge to develop a better set of structures for the data. This finally allows me to add some of the extra things that I’ve been struggling with – hopefully in a manner that also allows for better maintenance and easier upgrades!

I’ve made a good start on getting things changed and have a lot of the new design work done, so it’s just a question of finding and correcting all the small (and some not so small) bugs before pushing it live. I’m hoping to get that done this week, but it may drift into next week.

Shalin Shekhar Mangar: Resuming blogging

2013-06-03T19:42:00+00:00

I realize that it’s been a long time since I posted something here. About three years ago, I started working on projects at AOL, which did not have anything to do with Apache Solr. Increasingly I found myself having nothing to say publicly about my work. Though I did get back to working with Apache Solr for the AOL WebMail team, I didn’t resume blogging due to sheer laziness, I guess. (Yes, they use Solr! In fact, AOL WebMail just upgraded to Apache Solr 4.2.1 with impressive results!)

A lot has happened in the meantime. In December 2012, I joined the impressive team at LucidWorks — the Lucene/Solr company, to work (almost) full-time on open source search. In January, this year, I found my life partner and got married.

Now that I work on Lucene/Solr again, I think it is time to resume blogging regularly. I intend to write about new features, tutorials, tips & tricks and perhaps also explain the internals of Lucene/Solr features in greater depth. Here’s to a new start!

Claus Ibsen: See you soon at CamelOne 2013

2013-06-03T14:30:16+00:00

So CamelOne 2013 is starting on sunday evening with a reception with drinks and beers. What a great way to start. You can find details when and where at the agenda from the website. So make sure to attend on sunday evening if you can.

The conference then follows on monday and tuesday. The full agenda is up on the website at

This is the 3rd annual CamelOne conference, which is focused on the Apache integration projects such as Apache Camel, ActiveMQ, ServiceMix, Karaf and CXF. The speakers at the conference includes the founders of these projects, leaders, committers, long term end users, book authors, and as well end users who are using these projects in their companies.

We have a mix of speakers who are the people who hack on the projects all day long, and as well people who have used these projects in large projects at their companies. So you get information from both sides of the fence.

Rob Davies have blogged about what to look forward to at the conference. And as well giving a teaser for his talk about connecting applications everywhere with ActiveMQ.

Christian Posta who has dived deep into the internal of ActiveMQ is giving a talk about ... well the internals of ActiveMQ.

I will give a getting started with Apache Camel presentation, which is aimed for beginners or new users of Camel. And on the 2nd day I give a rundown of the common Camel components, and as well how to build your custom components. Both my talks is a mix with slides and practical live coding sessions.

I am much looking forward to James Strachan's ending keynote on the first day about using the hawtio web console with Apache Camel. There is gonna be some cool stuff revealed for the first time :)

Piergiorgio Lucidi: Drools Starter Review

2013-06-03T12:36:40+00:00

Packt Publishing has sent me a copy of this starter book that introduces the world of Drools.

Ruwan Linton: New in UltraESB 2.0 - Metrics and Monitoring

2013-06-03T08:06:04+00:00

Engineering team at AdroitLogic is very busy in delivering few new amazing things in the next couple of months, which includes the UltraESB 2.0 release with many new features and enhancements.

One major feature that we are planning to include is the metrics, monitoring and alerting support developed with the feedback from our users/customers. Let me give you some insight on what is coming up on that front!

These are some system monitoring graphs that your next version of UConsole will offer for the monitoring (still WIP)

Again system-wide, UltraESB elements monitoring such as the work managers and file caches and the total messages received and sent by the system

Proxy service specific metrics and graphs

Sequence statistics in the graphical format

Endpoint related metrics

On a separate Note: these snaps are from a stress/battle test suite that we have been running through the UltraESB for few days, to test the stability over the HTTP transport with the new Http-Core version, yes it is production ready even now, but we have few more touch-ups on the Management Console (UConsole), and a related product that we will be announcing with the UltraESB 2.0

So, this is just a single feature that we have included into the UltraESB 2.0 release, and there are lot more new features and enhancements to the UltraESB alone. Stay tuned, our release cycle will probably start on the month of August.

Rich Bowen: Mac OS X Time Machine disks on Linux

2013-06-03T00:02:22+00:00

Useful information about accessing a Mac OS X Time Machine backup disk on Linux: http://hints.macworld.com/article.php?story=20080623213342356

The relevant part being that if you look at a directory entry on the Time Machine disk, from Linux, you'll see something like:

-r--r--r-- 2155704 root 5791966 0 2007-06-25 02:54 Wallpaper

that first number, which is usually an indication of how many files are in a directory, is instead being used as a pointer to a directory named (in this case), dir_2155704 in the directory '.HFS+ Private Directory Data', which actually contains the files.

This information has proved very helpful over the last few days - although what I ended up doing was plugging the drive into a Mac and scp'ing the data across the network, since that was less hassle.

Looks like a good place for some Linux utility that provides a file browser for this kind of file system. Although I expect there's probably already one out there that I just didn't find yet.

Johannes Geppert: Struts2 jQuery Plugin 3.6.0 and Struts2 Bootstrap Plugin 1.6.1 available

2013-06-02T14:16:50+00:00

Struts2 jQuery Plugin

This release is mostly a maintenance release, this means it includes a new jQuery, jQuery UI and related Plugins Software Stack but also some Bug Fixes. For a complete list of changes since last version, please visit the Changelog.

The new release is based on jQuery UI 1.10.3 and includes some API Changes again. If you are upgrading from a previous Struts2 jQuery Version please respect the Upgrade Guide.

Library Updates

Download: http://code.google.com/p/struts2-jquery/downloads/list
Showcase: http://struts.jgeppert.com/struts2-jquery-showcase/
Issues: http://code.google.com/p/struts2-jquery/issues/list
User Group: http://groups.google.com/group/struts2-jquery

Struts2 Bootstrap Plugin

The new release 1.6.1 includes a patch provided by Brendan Le Ny. This patch allows an easy to use input prepend/append icon.

<s:textfield
        label="Textfield with prepended icon"
        inputPrependIcon="envelope"
        name="prependIcon"/>
 
<s:textfield
        label="Textfield with appended icon"
        inputAppendIcon="user"
        name="appendIcon"/>

Download: http://code.google.com/p/struts2-bootstrap/downloads/list
Showcase: http://struts.jgeppert.com/struts2-bootstrap-showcase/
Issues: http://code.google.com/p/struts2-bootstrap/issues/list

Rob Davies: Connecting Applications Everywhere with ActiveMQ

2013-05-31T11:52:06+00:00

This year at CamelOne there's going to be some exceptional presentations, but I'm also presenting "Connecting Applications Everywhere with ActiveMQ".

The focus of this presentation is to demonstrate the many protocol options and deployment scenarios that are available to Apache ActiveMQ. After an introduction to the Apache ActiveMQ project, and presenting why the "Internet of Things"is going to be driving the agenda for integration and messaging over the next 5 years, I'll be demonstrating an example application, going from an Arduino Microprocessor using MQTT to a MQTT/AMQP ActiveMQ gateway and to an ActiveMQ broker that will service HTML clients over WebSockets - something like this:

With so many linked components in a live demo - what could go wrong ? ;)

I'm hoping to catch up with lots of folks at CamelOne - if you haven't registered, its not too late!

Ioannis Canellos: Wicket, Spring 3, JPA2 & Hibernate OSGi Application on Apache Karaf

2013-05-30T16:01:16+00:00

EDIT: Hibernate is now OSGi ready so most of those stuff are now completely outdated.

The full source for this post has been move to github.

Prologue
Recently I attempted to modify an existing crud web application for OSGi deployment. During the process I encountered a lot of issues such as

Lack of OSGi bundles.
Troubles wiring the tiers of the application together.
Issues on the OSGi container configuration.
Lack of detailed examples on the web.

So, I decided to create such a guide & provide full source for a working example (A very simple person crud application).

The first part of this guide is Creating custom Hibernate 3.5 OSGi bundles. This part provides an example project (which includes the bundles source) that describes how to use the custom hibernate bundles in order to build a wicket, spring 3, hibernate 3.5 jpa 2 and deploy it to Karaf.

Among others it describes:

How to wire database and web tier using the OSGi blueprint.
How to deploy web applications to Karaf 1.6.0.
A small wicket crud application.

Note: This demo application does not make use OSGi Enterprise Spec, since its an OSGi-fication of an existing application. The use of the spec will be a subject for future posts.

Enjoy!

Environment Preparation
The OSGi run-time that will be used in this post is Felix/Karaf version 1.6.0.
This section describes the required configuration for deploying web applications.

Once, karaf is downloaded and extracted, it can be started by typing

bin/karaf

from inside the karaf root folder.

Now, we are going to install karaf webconsole and war deployer that will allow us to deploy web applications to karaf.

features:install webconsole
features:install war

Note: In the background karaf fetches all the required bundles from maven repositories. You are going to need internet access for this. Moreover, if you are behind a proxy you will need to set up your jvm net.properties accordingly. Having the proxy configured in maven settings.xml is not enough.

Custom Bundles
Most of the bundles required for this project are available either in public maven repositories or inside Spring Enterprise Bundle Repository. However, hibernate 3.5.x which is one of the key dependencies for this project is not available as OSGi bundle (note: earlier version of hibernate can be found in Spring EBR). More details on OSGi-fying Hibernate 3.5.x in the previous part of the guide "Creating custom Hibernate 3.5 OSGi bundles".

Creating the application itself
The actual demo application will be the simplest possible wicket crud for persons (a killer application that stores/delete/updates a persons first name and last name to the database).

Database
The create schema script of such application in mysql would look like this:

CREATE TABLE person (
    ID MEDIUMINT NOT NULL AUTO_INCREMENT,
    FIRST_NAME VARCHAR(40) NOT NULL,
    LAST_NAME VARCHAR(40) NOT NULL,
    PRIMARY KEY (ID)
);

Database Tier
For the database tier we are going to create a simple bundle that will contain the entity, the dao interface and the dao implementation. The bundle will contain the necessary persistence descriptor for JPA 2.0 with hibernate as persistence provider. Finally it will use spring to create the data source, entity manager factory & JPA transaction manager. This bundle will export the dao as a service to the OSGi Registry using Spring dynamic modules.

The Person entity for the example can look like:

package net.iocanel.database.entities;

import java.io.Serializable;
import javax.persistence.Basic;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.NamedQueries;
import javax.persistence.NamedQuery;
import javax.persistence.Table;

/**
 *
 * @author iocanel
 */
@Entity
@Table(name = "Person")
@NamedQueries({
 @NamedQuery(name = "Person.findAll", query = "SELECT p FROM Person p"),
 @NamedQuery(name = "Person.findById", query = "SELECT p FROM Person p WHERE p.id = :id"),
 @NamedQuery(name = "Person.findByFirstName", query = "SELECT p FROM Person p WHERE p.firstName = :firstName"),
 @NamedQuery(name = "Person.findByLastName", query = "SELECT p FROM Person p WHERE p.lastName = :lastName")})
 public class Person implements Serializable {

 private static final long serialVersionUID = 1L;
 @Id
 @GeneratedValue(strategy = GenerationType.AUTO)
 @Basic(optional = false)
 @Column(name = "ID")
 private Integer id;
 @Column(name = "FIRST_NAME")
 private String firstName;
 @Column(name = "LAST_NAME")
 private String lastName;

 public Person() {
 }

 public Person(Integer id) {
  this.id = id;
 }

 public Integer getId() {
  return id;
 }

 public void setId(Integer id) {
  this.id = id;
 }

 public String getFirstName() {
  return firstName;
 }

 public void setFirstName(String firstName) {
  this.firstName = firstName;
 }

 public String getLastName() {
  return lastName;
 }

 public void setLastName(String lastName) {
  this.lastName = lastName;
 }

 @Override
 public int hashCode() {
  int hash = 0;
  hash += (id != null ? id.hashCode() : 0);
  return hash;
 }

 @Override
 public boolean equals(Object object) {
  // TODO: Warning - this method won't work in the case the id fields are not set
  if (!(object instanceof Person)) {
   return false;
  }
  Person other = (Person) object;
  if ((this.id == null && other.id != null) || (this.id != null && !this.id.equals(other.id))) {
   return false;
  }
  return true;
 }

 @Override
 public String toString() {
  return "net.iocanel.database.entities.Person[id=" + id + "]";
 }
}

For this entity we will create a dao interface, through which the rest of the bundles in the container can track/lookup the dao service (the actual implementation).

We want the dao service to provide simple crud operations such as, create, delete, find & findAll, so the dao interface can be something like:

package net.iocanel.database.dao;

import java.util.List;
import net.iocanel.database.entities.Person;

/**
 *
 * @author iocanel
 */
public interface PersonDAO {

 public void create(Person person) throws Exception;
 public void edit(Person person) throws Exception;
 public void destroy(Integer id) throws Exception;
 public Person findPerson(Integer id);
 public List findAllPersons();
}

The actual jpa implementation of the dao will obtain the EntityManager via Spring (it will be injected by Spring) and for transaction demarcation will use Spring's Transactional annotation:

package net.iocanel.database.dao;

import java.util.List;
import javax.persistence.EntityManager;
import javax.persistence.Query;
import javax.persistence.PersistenceContext;
import net.iocanel.database.entities.Person;
import org.springframework.transaction.annotation.Transactional;

/**
 *
 * @author iocanel
 */
@Transactional
public class PersonJpaDAO implements PersonDAO {

 @PersistenceContext
 private EntityManager entityManager;

 public void create(Person person) throws Exception {
  entityManager.persist(person);
  entityManager.flush();
 }

 public void edit(Person person) throws Exception {
  entityManager.merge(person);
  entityManager.flush();
 }

 public void destroy(Integer id) throws Exception {
  entityManager.remove(findPerson(id));
  entityManager.flush();
 }

 public List findPersonEntities(int maxResults, int firstResult) {
  return findPersonEntities(false, maxResults, firstResult);
 }

 private List findPersonEntities(boolean all, int maxResults, int firstResult) {
  Query q = entityManager.createQuery("select object(o) from Person as o");
  if (!all) {
   q.setMaxResults(maxResults);
   q.setFirstResult(firstResult);
  }
  return q.getResultList();
 }

 public Person findPerson(Integer id) {
  return entityManager.find(Person.class, id);
 }

 public int getPersonCount() {
  Query q = entityManager.createQuery("select count(o) from Person as o");
  return ((Long) q.getSingleResult()).intValue();
 }

 public List findAllPersons() {
  Query q = entityManager.createNamedQuery("Person.findAll");
  return q.getResultList();
 }
}

For the EntityManager injection and Spring Transactions, we need need a spring context. Since we are going to use Spring Dynamic Modules, the spring context needs to be placed under META-INF/spring/.

For the creation of the EntityManagerFactory Spring will need a persistence.xml file located under META-INF:


  
    org.hibernate.ejb.HibernatePersistence
    net.iocanel.database.entities.Person

So far in the database tier we did what we would do in a typical application. Now we will add OSGi flavor to our module.

Creating the DAO OSGi Service
As mentioned above for the creation of the dao service we will use spring dynamic modules. So all we need is to add a descriptor under META-INF/spring that will instruct Spring's OSGi exporter to export bean personDAO as OSGi service:

Finally, we need to perform a small hack. In the previous part of this guide, we created an OSGi fragment for Hibernate Validator. This fragment is attached on the validation api host, so that the validation api can find the classes of hibernate validator. However, we still need to instruct the validation api, to look for Hibernate Validator classes. In an non-OSGi world the validation api will lookup in the classpath for the following file META-INF/services/javax.validation.spi.ValidationProvider and read the actual validation provider class name from this file.

Passing the Validation Provider to Validation API
In the OSGi world the validation api, will delegate that call to the calling bundle (in our case the database tier bundle) so we are going to make sure that it finds it. How we are going to do so? We are going to copy it from Hibernate Validator and add it in our bundle. This approach might not seem that elegant, however it has two great advantages:

Its simple
It works

If you are aware of more elegant alternative feel free to communicate them.

The final obstacle is creating the bundle itself.The bundle will be created using maven-bundle-plugin. As maven dependencies it will contain only whatever it requires for the compile scope and its run-time dependencies(hibernate,spring,jpa spec, cglib etc) will be declared as OSGi Import-Packages.


  net.iocanel.*
  
   *,
   javax.sql,
   javax.transaction,
   javax.persistence,
   javax.persistence.*,
   javax.persistence.spi,
   javax.persistence.metamodel,
   javax.persistence.criteria,
   org.hibernate,
   org.hibernate.annotations,
   org.hibernate.annotations.common,
   org.hibernate.annotations.common.reflection,
   org.hibernate.ejb,
   org.hibernate.cfg,
   org.hibernate.cfg.annotations,
   org.hibernate.cfg.annotations.reflection,
   org.hibernate.cache,
   org.hibernate.hql,
   org.hibernate.hql.ast,
   org.hibernate.validator,
   org.hibernate.validator.constraints,
   org.hibernate.validator.constraints.impl,
   org.hibernate.validator.engine,
   org.hibernate.validator.engine.groups,
   org.hibernate.validator.engine.resolver,
   org.hibernate.validator.jtype,
   org.hibernate.validator.metadata,
   org.hibernate.validator.util,
   org.hibernate.validator.util.annotationfactory,
   org.hibernate.validator.xml,
   org.hibernate.proxy,
   com.mysql.jdbc,
   javassist.util.proxy,
   org.aopalliance.aop,
   org.springframework.aop,
   org.springframework.aop.framework,
   net.sf.cglib.core,
   net.sf.cglib.reflect,
   net.sf.cglib.proxy
  
  !*
  ${pom.groupId}.${pom.artifactId}
  false
 

 
  
   
    org.apache.felix
    maven-bundle-plugin
    true
    
     
      ${artifactId}
      ${symbolic.name}
      ${pom.name}
      .
      ${import.packages}
      ${export.packages}
      ${private.packages}

Presentation/Web Tier
For the presentation tier we are going to be a Wicket OSGi application. This application will be integrated with Spring using @SpringBean annotation (more details on this on Wicket/Spring Wiki).

Since we are interested in taking advantage of Spring Dynamic Modules, we are going to instruct to load its context from OsgiBundleXmlWebApplicationContext inside the web.xml.


  contextClassorg.springframework.osgi.web.context.support.OsgiBundleXmlWebApplicationContext

So the full web deployment descriptor for Wicket/Spring/OSGi could look like this(yes I know I am starting to sound like Bob Ross):



 web-tier

 
  wicket.configurationdevelopment

 
  contextClassorg.springframework.osgi.web.context.support.OsgiBundleXmlWebApplicationContext

 
  org.springframework.web.context.ContextLoaderListener
 
 
  wicket.wicket-spring
  org.apache.wicket.protocol.http.WicketFilter
  
   applicationFactoryClassNameorg.apache.wicket.spring.SpringWebApplicationFactory
  
   applicationClassNamenet.iocanel.WicketApplication
 
 
  wicket.wicket-spring
  /*

The Spring context file (/WEB-INF/applicationContext.xml)that will be loaded needs to define two simple things:

The Wicket Application Object.
The PersonDAO OSGi service.

The PersonDAO service will be looked up using Spring Dynamic Modules. Inside the wicket application the PersonDAO service will be injected as if it was a normal spring bean using the @SpringBean annotation.

We are almost there. All that's left is the coding of the actual crud. I will not go into much detail, since its beyond the scope of this blog post. However, I am going to list the key points of the crud.

The C.R.U.D.
For the CRUD part we will create a single ajax page that will display:

A list of all persons in the database.
A small form to insert/edit person details.
Buttons for each record to edit/remove persons in the database.

The List Component that will be used is PropertyListView, and the model attached to list will be a LoadableDetachableModel that will load all persons from the database. Finally the person details form will consist of 2 text fields First Name & Last Name.

Enjoy!

The full source for this example (including the custom bundles) can be found at my github repository. Once you unpack it you can mvn clean install and it will package the project bundles, the custom bundles and all the required bundles under target/wicket-osgi.dir/deploy folder. Just copy the contents of this folder to $KARAF_HOME/deploy and you are ready launch the application at http://localhost:8181/web-tier/.

Final thoughts
I hope you find this useful.
Once my schedule allows, I will blog on how to add JTA transactions on the example above, so stay tuned(The Hibernate bundle is JTA ready, however we need a JTA transaction manager bundle).
Feel free to send comments or suggestions.

Ioannis Canellos: Creating custom Hibernate OSGi bundles for JPA 2.0

2013-05-30T16:00:31+00:00

Edit: I am more than happy that this post is now completely obsolete. Hibernate is now OSGi ready, Yay!

Prologue
I was trying to migrate an application that uses JPA 2.0 / Hibernate to OSGi. I found out that hibernate does not provide OSGi bundles. There are some Hibernate bundles provided in the Spring Enterprise Bundle repository, however they are none available for Hibernate 3.5.x which implements JPA 2.0. So I decided to create them myself and share the experience with you.

This post describes how to OSGi-fy Hibernate 3.5.2-Final with EhCache and JTA transaction support. The bundles that were created were tested on Felix Karaf, but they will probably work in other containers too.

Introduction
A typical JPA 2.0 application with Hibernate as persistence provider will probably require among other the following dependencies

hibernate-core
hibernate-annotations
hibernate-entitymanager
hibernate-validator
ehcache

Unfortunately, at the time this post was written none of the above was available as OSGi bundle. To make OSGi bundles for the above one needs to overcome the following problems

Cyclic dependencies inside Hibernate artifacts.
3rd party dependencies (e.g. Weblogic/Websphere Transaction Manager).
Common api / impl issues for validation api and hibernate cache.

The last bullet which may not be that clear points to a problem where an api loads classes from the implementation using Class.forName() or similar approaches. In the OSGi world that means that the api must import packages from the implementation.

Hibernate cyclic dependencies
The creation of an OSGi bundle for each hibernate artifact is possible. However, when the bundles get deployed to an OSGi container, they will fail to resolve due to cyclic package imports.

The easiest way to overcome this issue is to merge hibernate core artifacts into one bundle. Below I am going to provide an example of how to use maven bundle plug-in to merge hibernate-core, hibernate-annotations & hibernate-entitymanager into one bundle.

A common way to use the maven-bundle-plugin to merge jars into artifacts is to instruct it to embed the dependencies of a project into a bundle. However, this is not very handy in cases where you need to add custom code into the final bundle. In that case you can use the maven dependency plug-in to unpack the dependencies, bundle plug-in to create the manifest and jar plug-in to instruct it to use the generated manifest in the package phase.


  
   org.apache.maven.plugins
   maven-dependency-plugin
   
    
     unpack
     generate-resources
     
      unpack
     
     
      
       
        org.hibernate
        hibernate-core
        false
        target
       
       
        org.hibernate
        hibernate-annotations
        false
        target
       
       
        org.hibernate
        hibernate-entitymanager
        false
        target
       
      
      false
      true
     
    
   
  
  
   org.apache.felix
   maven-bundle-plugin
   2.0.1
   true
   
    
     create-manifest
     process-classes
     
      manifest
     
    
   
   
    
     jar
     bundle
    
    ${unpack.bundle}
    
     ${artifactId}
     ${groupId},${artifactId}
     ${project.name}
     *
     *
    
   
  
  
   maven-jar-plugin
   
    
     target/META-INF/MANIFEST.MF

Hibernate & 3rd Party dependencies
Hibernate has a lot of 3rd party dependencies. Some of them are available as OSGi bundles, some need to be created and some can be excluded.

Examples of 3rd party dependencies that are available as OSGi bundle in the Spring Enterprise Repository are:

antlr
dom4j
cglib

Dependencies that are not available are:

jacc (javax.security.jacc)

Dependencies that can be excluded vary depending on the needs. In my case I could exclude Weblogic/Websphere transaction manager, since I didn't intend to use them. To exclude a dependency just add the packages that are to be excluded in the import packages section using the ! operator (e.g. !com.ibm.*,*)

Hibernate validator and Validation API
As mentioned above the validation api provides a factory that build the validator by loading the implementing class using Class.forName(). This issue can be solved with 2 ways

Use dynamic imports in the API bundle to import the Implementation at runtime.
Make the implementation OSGi Fragment that will get attached to the API.

In this example the validation api is the one provided by the Spring Enterprise Bundle Repository, so the second approach was easier to apply.

More details on this issue can be found at this excellent blog post:
Having “fun” with JSR-303 Beans Validation and OSGi + Spring DM

Hibernate & EhCache
More or less the same applies to EhCache. Hibernate provides an interface which is implemented by EhCache. Hibernate loads that implementation in runtime. We will do exactly the same thing we did for hibernate validator. We will convert ehcache jar to fragement bundle so that it gets attached to the merged hibernate bundle.

Hibernate & JTA Transactions
I kept for last the most interesting part. This part describes what needs to be added inside the bundle so that it can support JTA transactions.

For JTA transactions Hibernate needs a reference to the transaction manager. That reference is returned by the TransactionManagerLookup class specified in the persistence.xml. In a typical JEE container the lookup class just performs a JNDI to get the TransactionManager. In an OSGi container the transaction manager is very likely to be exported as an OSGi service.

This section describes how to build an OSGi based TransactionManagerLookup class. The solution presented is very simple and uses only the OSGi core framework (no blueprint implementation required).

We will add to the hibernate bundle 3 new classes:

TransactionManagerLocator (Service Locator).
OsgiTransactionManagerLookup (Lookup implementation).
Activator (Hibernate Bundle Activator).

TransactionManagerLocator is a ServiceLocator that uses OSGi's ServiceTracker to get a reference to the TransactionManager service.

package org.hibernate.transaction;

import javax.transaction.TransactionManager;
import org.osgi.framework.BundleContext;
import org.osgi.util.tracker.ServiceTracker;

/**
 *
 * @author iocanel
 */
public class TransactionManagerLocator {

    private final String lookupFilter = "(objectClass=javax.transaction.TransactionManager)";
    private static BundleContext context;
    private static TransactionManagerLocator instance;
    private ServiceTracker serviceTracker;

    //Constructor
    private TransactionManagerLocator() throws Exception {
        if (context == null) {
            throw new Exception("Bundle Context is null");
        } else {
            serviceTracker = new ServiceTracker(context, context.createFilter(lookupFilter), null);
        }
    }

    public static synchronized TransactionManagerLocator getInstance() throws Exception {
        if (instance == null) {
            instance = new TransactionManagerLocator();
        }
        return instance;
    }


    public static void setContext(BundleContext context) {
        TransactionManagerLocator.context = context;
    }

    public TransactionManager getTransactionManager() {
        return (TransactionManager) serviceTracker.getService();

    }
}

OsgiTransactionManagerLookup is an implementation of Hibernates TransactionManagerLookup that delegates the look
up to the TransactionManagerLocator.

package org.hibernate.transaction;

import java.util.Properties;
import javax.transaction.Transaction;
import javax.transaction.TransactionManager;
import org.hibernate.HibernateException;

/**
 *
 * @author iocanel
 */
public class OsgiTransactionManagerLookup implements TransactionManagerLookup {

    public TransactionManager getTransactionManager(Properties props) throws HibernateException {
        try {
            return TransactionManagerLocator.getInstance().getTransactionManager();        } catch (Exception ex) {
            throw new HibernateException("Failed to lookup transaction manager.", ex);
        }
    }

    public String getUserTransactionName() {

        return "java:comp/UserTransaction";
    }

    public Object getTransactionIdentifier(Transaction transaction) {
        return transaction;
    }
}

Activator is just a bundle activator. Its role is to pass a static reference of the bundle context to the TransactionManagerLocator (the bundle context is required by the service tracker).

package org.hibernate;

import org.hibernate.transaction.TransactionManagerLocator;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;

/*
 *
 * @author iocanel
 */
public class Activator implements BundleActivator {

    public void start(BundleContext bc) throws Exception {
        TransactionManagerLocator.setContext(bc);
    }

    public void stop(BundleContext bc) throws Exception {
    }
}

Example use of the bundle & bundle source code.
An example web application that uses the custom hibernate bundles can be found in this post.

If you feel tired of reading and just want to use the bundles. You can download them from here. All the custom bundles are included in the maven project under the bundles folder (as seen in the picture).

The example application uses Wicket and can be easily deploy in Karaf.
Final Thoughts
I hope you found it useful.

Any feedback is more than welcome.

Tim Bish: Win A Free Copy of Packt's Instant Apache ActiveMQ Messaging Application Development How-to!!

2013-05-30T15:28:12+00:00

I am pleased to announce that I have teamed up with Packt Publishingand we are organizing a giveaway especially for you. All you need to do is just comment below the post and win a free e-copy ofInstant Apache ActiveMQ Messaging Application Development How-to. Two lucky winners stand a chance to win an e-copy of the book. Keep reading to find out how you can be one of the Lucky One.

Overview of Instant Apache ActiveMQ Messaging Application Development How-to

Learn something new in an Instant! A short, fast, focused guide delivering immediate results.
Learn how to use the JMS API
Explore advanced messaging features in ActiveMQ
Useful information on common pitfalls new developers often encounter

How to Enter?

Simply post your expectations from this book in comments section below. You could be one of the 2 lucky participants to win the copy.

DeadLine:

The contest will close on 06/14/2013 . Winners will be contacted by email, so be sure to use your real email address when you comment!

Danny Angus: note to self, how to fix "requires installation of untrusted packages" in Ubuntu

2013-05-30T10:44:13+00:00

I tried to install a couple of updates on my work laptop, running Ubuntu 12.04, a minute or two ago when prompted by the update manager, and I got the "requires installation of untrusted packages" error. I've since dicovered that there are two ways round this, one is to allow untrusted packages by doing the update as root or via sudo, and saying "y" when asked

Install these packages without verification [y/N]?

But I don't like that, you don't know who might be spoofing you with what fake packages. The better way is to rebuild the apt cache. I found the instructions here, I'm claiming no credit for solving the problem, just reproducing them below so that I can be sure to find them again next time I need them. Hopefully some of you who read this in the future will also benefit.

sudo apt-get clean
cd /var/lib/apt
sudo mv lists lists.old
sudo mkdir -p lists/partial
sudo apt-get clean 
sudo apt-get update

Enjoy!

Chip Childers: Inaugural CloudStack DC and NoVA User Group

2013-05-29T17:00:58+00:00

I'm about to speak at the CloudStack DC and NoVA User Group's innagural meetup, which the folks at Basho were kind enough to make a joint-meetup with the Riak DC group. Registration between the two groups (roughly deduped) is at around 70 people for the event. This should be a fun evening!

For those of you that can't make it, here's what I'm presenting:

CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview from Chip Childers

Rob Weir: Who wants to develop OpenOffice for Tablet?

2013-05-29T15:24:54+00:00

One of the most common user questions I see on the Facebook and Twitter streams for Apache OpenOffice is “Do you have a iPad version?” or “Do you have a tablet version”? Although there are companies that offer access to OpenOffice via a virtualized remote session, there is no native tablet version of OpenOffice.

I have received questions, behind the scenes, about the feasibility of starting such an effort at Apache. Of course, creating a tablet version of OpenOffice, a competitive application with a first-class native touch UI, with platform integration and optimization, is a non-trivial effort. My impression is that there are several companies, small and large, that would find this to be an intriguing possibility. But the task is too large to do it alone. But with several companies involved, as a joint effort, in an open source project, then this becomes possible.

Imagine if we had such an open source tablet version of OpenOffice available today. It would be an app that everyone would want. If done right the OpenOffice app would be at the top of the charts just as the desktop OpenOffice is one of the leading open source desktop apps. The app itself would be free, of course. But it would be an open platform that we could all build upon.

Possible business models might include:

Cloud services related to documents, range from storage to sharing and collaboration
Extensions to the app, in-app purchases of additional templates, content, etc.
Advertising-supported apps.
From service provider perspective, avoidance of licensing fees for competing commercial office software.
A “white label” version that can be rebranded per customer

There are good reasons, I think, for doing such work at Apache, including:

Existing expertise in the OpenOffice product
Proven community development culture based on The Apache Way
Permissive, commercially-friendly Apache License, the preferred license for Android userspace
Strong brand / name recognition

I’d like to have a discussion with those having a serious interest in making a tablet version of OpenOffice. By serious, I mean those who might be willing to contribute developers to a larger effort, if such an effort were to materialize. I’m happy to talk one-on-one. And if there is sufficient serious interest from multiple parties I can broker a meeting of interested parties to discuss further options.

If any of this sounds interesting and you want to register your interest please send me an email at robert_weir@us.ibm.com.

Edward J. Yoon: Keep your eyes open

2013-05-29T11:46:05+00:00

만약 당신이 소프트웨어 개발이든 뭐든 자기 분야에 자신이 있으면, 굳이 빌어(?)먹을 필요가 없다! 당장의 현실적인 문제는 핑계에 불과하다.

직장생활이든 뭐든 어떤 조직에 몸을 담게되면, 나도 모르는 사이 남이 만들어놓은 울타리 안에서 자신의 삶을 꾸리게 되는데, 이러한 상태, 즉, 시야가 굉장히 좁아진 우물안 개구리 상태에서는 누구나 울타리 밖이 두려울수밖에. 하지만, 내가 자신있게 얘기할 수 있다.

자신있으면 쫄지말고 실천하라! 자신이 하이에나 새끼인지 호랑이 새끼인지는 일단 이빨을 드러내고 육식을 시작하라 ㅋ.

Claus Ibsen: Long stacktraces in Apache Camel no more

2013-05-29T06:56:24+00:00

In the upcoming Apache Camel 2.12 we have improved the internal routing engine to dramatically reduce the stacktraces, our end users will see when an exception occurred and was logged.

During the development of the improvement we had a small sample route for benchmarking.
It was basically a simple route with:

from("seda:start")
.to("log:foo")
.to("log:bar")
.to("log:baz")
.process(new Processor ...);

And in the processor we forced an exception being logged. In Apache Camel 2.11.x the stacktrace would be around 40 lines. The Camel team managed to reduce this down to 15 lines. And the bottom 6 lines is 3 from the JVM and 3 from the seda consumer. So in between the routing engine is executing; and that's where we have optimized the code. So we went from (40-6 = 34) to (15-6 = 9). Eg before we had 34 lines during the routing, and that is down to 9 now.

Camel is still executing the same functionality. As an end user there is no functionality turned off or missing now. Its all still there and being executed; just in a different way. What we did was refactor most of the internal cross cutting functionality into a single entity CamelInternalProcessor (name subject for change), and fold that into a single stack-frame being executed.

This improvement is internal changes only, and all end users Camel application is upgrade compatible.

For our end users who have been debugging the Camel source code, this is now also simpler, as we added code comments in the CamelInternalProcessor with pointers how to do this easier, and which part you can safely skip over, to make your debugging experience easier and faster.

And speaking of debugger. Then we also added a new BacklogDebugger, which is a Camel routing debugger JMX MBean which is ready to use for tooling and end users. For example you can use it easily set breakpoints, single step a message in the routes, and edit/view its message content. Now for the graphical UI, then James Strachan is hacking on hawtio to add this in the upcoming hawtio 1.2 release. And since its JMX based you can always use it from a JMX console such as JConsole or JVisualVM. And for the brave a plugin to Eclipse / IDEA could be developed to integrate it with their native debugger; but thats a big task to take on.

For further details and background on the stacktrace improvement then you can take a look at the ticket CAMEL-6077 and our discussion on the Camel developer mailing list. As well we have a sample of the before and after stacktraces in the CAMEL-6077 ticket.

Tim Bish: New ActiveMQ Book Published.

2013-05-28T22:42:15+00:00

Recently finished work on a new book for the beginning ActiveMQ user, "Instant Apache ActiveMQ Messaging Application Development How-to" published by Packt Publishing. The book is a fast paced walk through as the basic skills needed to work with Apache ActiveMQ. At 78 pages its a quick read and provides some helpful tips for some of the most common questions asked by those new to working with ActiveMQ.

Some things that are covered in this book.

Install and run an ActiveMQ Broker
Set up a basic application developer environment for ActiveMQ using Apache Maven
Write basic JMS applications using topics and queue to broadcast events
Perform Request and Response style operations over JMS
Write applications that embed ActiveMQ brokers directly into their own JVM
Utilize ActiveMQ's fault tolerant features to develop robust client applications
How to delay message sending using ActiveMQ's built in scheduler

The book is available in eBook form from Packt or you can also order a paperback copy on Amazon.

Chip Childers: CloudStack 4.1.0 Voting Restarted

2013-05-28T17:41:16+00:00

A month ago, I noted that the Apache CloudStack project was taking it's time to ensure that upgrades were working well for our 4.1.0 release. We have sorted out quite a few issues over the last month, and are now proceeding with a new voting round. While it's possible that a new blocker will be found, we are back on track to getting the release out the door. Feel free to subscribe to the announce@cloudstack.apache.org mailing list (send an email to announce-subscribe@cloudstack.apache.org to subscribe) to get the announcement email when we have everything ready to go!

Jeremy Quinn: Fresh [Flickr]

2013-05-28T17:40:13+00:00

sharkbait posted a photo:

I don't know what this one is called she makes lovely flowers as well as theSE tiny fresh spring leaves.

Community Over Code: Congratulations to the 2013 Apache Board of Directors

2013-05-28T14:30:10+00:00

The ASF recently held it’s Annual Member’s Meeting where all Members of the Foundation cast ballots in the annual election for the Board. I was lucky enough to be elected, so I will be returning to the board, along with new first time Director Chris Mattmann. Everyone also thanked our two outgoing Directors, Rich Bowen and Ross Gardler.

The new board comprises:

Shane Curcuru
Doug Cutting (chairman)
Bertrand Delacretaz
Roy T. Fielding
Jim Jagielski
Chris Mattmann
Brett Porter
Sam Ruby
Greg Stein

I also keep a graphical history of the Apache board.